Frameworks are workload assumptions
An argument for bespoke code.
TL;DR
- Frameworks encode assumptions about workloads. When those assumptions stop fitting, the framework works against you.
- Bespoke paths do not have to stay bespoke. llama.cpp, vLLM, and SGLang show how workload-specific execution can become the next framework.
- Large-scale model workloads are repetitive and expensive enough that bespoke execution can make economic sense.
- Recent frontier models are optimizing their own execution stacks, substantially lowering the cost of bespoke code.
1. Measure before you cut
Figure 1. Giovanni Battista Moroni, The Tailor, c. 1565–1570. [source]
1.1 Bespoke and off the rack
Moroni’s tailor begins with one body. He measures it, marks the cloth, and cuts a garment for that person. Performance engineering can work the same way: benchmark one workload, then write the execution code around what was measured.
Off-the-rack clothing starts from a different premise. Standardized sizes will not fit anyone exactly, but they make clothing cheaper to produce and easier to distribute. Framework-first development makes the same trade.Here I use “framework” broadly for a system that asks a workload to adopt its representation and execution model. Examples include PyTorch, JAX/XLA, TensorFlow, Spark, and database query engines. The workload is expressed using standard operations and data structures. In return, the developer gets existing kernels, portability, debugging tools, and a shorter path to working software.
The difference is where the process begins. A bespoke implementation begins with the workload and derives an execution model from it. A framework-first implementation begins with an existing execution model and maps the workload onto it.
1.2 Why the old bargain changed
For decades, framework-first was usually the cheaper choice. Applications changed frequently, custom execution code required scarce engineering time, and general-purpose compute was relatively cheap. The famous advice against premature optimizationDonald Knuth, Structured Programming with go to Statements (1974). fit this bargain: get the program working, measure it, and optimize only when the cost becomes significant.
Traditional application stacks often coordinate a mixture of unrelated programs, each with different memory behavior and latency requirements. Their resources are shared across changing work, so the stack cannot assume much about what runs next.
Repeated model execution offers a narrower target. When a known architecture is trained or served at scale on GPU/NPU, the same tensor operations recur across batches or requests. A runtime can organize that work around throughput, queue compatible requests, and batch their execution; continuous batching in vLLM is one example. Training changes the weights and inference changes the runtime state, but the overall computation is comparatively stable.
LLM inference makes this especially clear. Since GPT-3 in 2020, dominant general-purpose LLMs have remained autoregressive transformers. Model details change, but serving still consists largely of compute-heavy prefill, bandwidth-heavy decode, and a growing KV cache.
This work is repeated at enormous scale. Its recurring cost is measured directly in tokens, and the surrounding investment in accelerators, power, and data centers shows the size of the expense. A small inefficiency is paid again for every generated token.
At this scale, engineering time is no longer obviously more expensive than compute. A month spent reducing memory traffic per token may cost less than running that traffic across a fleet.
2. The workload that did not fit
If you have worked on model optimization, you have probably fought the framework at some point. What should be a narrow change ends up crossing several layers of abstraction. I ran into this in 2020 while experimenting with low-bit transformers for on-device execution.
TensorFlow Lite was the most prominent on-device ML framework at the time. By then, transformers had been reshaping the state of the art across NLP since their introduction in 2017. My naive assumption was that transformer execution would already be supported, but autoregressive inference contradicted the workload assumptions built into TF Lite.There was also a significant feature gap even for recurrent models. As late as July 2020, TF Lite had only experimental LSTM support without quantization support.
For the path I was building, TF Lite’s graph represented a complete CNN inference well, but only one step of autoregressive inference. The loop and its growing state lived outside the abstraction: pre-allocating for the maximum sequence length wasted scarce mobile memory, while resizing weakened the static memory plan. Figure 2 breaks down the difference.Figure 2 uses a decoder-only model because that is now the dominant LLM architecture. The 2020 translation system also ran an encoder once before autoregressive decoding.
Figure 2. (Left) A complete CNN workload fits in a single TF Lite graph invocation, so tensor lifetimes are bounded and its memory can be preallocated once (orange dotted line). (Right) In the path I was building, a TF Lite graph invocation captured only one decoder step, while the complete autoregressive workload also included an outer token loop and a KV cache that persisted and grew beyond the same static allocation.
Despite the lack of support, framework-first was the default mindset, so I initially tried to implement the missing path inside TF Lite. The (attempted) implementation crossed TensorFlow graph tracing, converter and fusion passes, the serialized graph definition, runtime memory management, custom-operator registration, and the on-device kernels. Each layer had its own conventions, and a change at one layer could be transformed or rejected by the next.
In the end, I wrote the inference path directly in C++. It was more straightforward than I expected. At the time, my only C++ experience was from school. The test-driven implementation took two weeks. Most of the work was writing a numerically stable inference loop and checking each operation against dumped activation values. Writing it directly also made execution transparent and gave me control over memory allocation, tensor layout, and the quantized kernels. The result was a production-grade, extremely low-bit transformer running on-device (shameless plug, I know, but in 2020 it was the first transformer quantization system to run below 3 bits: paper, blog).
The implementation made the workload legible: the autoregressive loop, persistent state, memory layout, and quantized kernels were all explicit. Those details were too specific for the old framework, but they were not unique to my implementation.
3. When the workload becomes the framework
As transformer inference became common, the same constraints appeared everywhere. They became the assumptions of a new set of frameworks. llama.cpp packaged the inference path itself. vLLM and SGLang packaged the outer serving loop.
3.1 The inference path becomes a framework
llama.cpp implemented Llama inference directly in C/C++, around quantized weights, explicit KV state, and hardware-specific backends, rather than importing the model through a general ML framework. It has since grown from a narrow implementation into one of the main frameworks for local and on-device LLM inference.
3.2 The outer loop becomes a framework
vLLM and SGLang specialized a wider boundary. They formalized the outer loop of LLM serving while keeping lower-level model execution behind backend interfaces. The runtime could now see requests arriving at different times, the transition between prefill and decode, and the state carried across model invocations.
Where a general ML framework sees model invocations, vLLM sees a changing pool of requests. PagedAttention virtualizes the KV cache so blocks can be allocated and shared as sequences grow, instead of reserving one contiguous maximum-sized buffer per request. Continuous batching uses the same view to schedule work across requests.
Figure 3. PagedAttention maps a request’s contiguous logical KV blocks onto physical blocks allocated on demand. vLLM reports reducing KV-cache waste from 60 to 80 percent to under 4 percent, with 2 to 4 times the throughput of FasterTransformer and Orca at similar latency. [sources: vLLM, paged-attention]
SGLang also sees relationships between generation calls. Its radix-tree cache can reuse shared prefixes, turning repeated prompts and conversation histories into reusable execution state.
Section 2 showed how inherited framework assumptions can work against a workload. These examples show the other side: redrawing the framework boundary around the workload makes new optimizations possible.
4. The kernel boundary breaks down
The same pattern soon appeared one level lower. Transformer engines conventionally express a forward pass as a sequence of roughly operator-sized CUDA kernels. CUDA makes each launch a natural boundary for dependency ordering and grid-wide synchronization. Operator fusion can merge neighboring operations, but recent megakernels move much more of the workload and its coordination inside one persistent kernel.
Those boundaries are convenient: the next kernel waits for the work it depends on. The cost is visible between kernels, where launch overhead and straggling thread blocks can leave the GPU idle. For large batches, there is often enough work inside each kernel to amortize this overhead. For low-latency, batch-one decoding of a small model, there may not be.
Figure 4. The red boxes show kernel boundaries around the operations in a Llama-1B block. In the No Bubbles authors’ words, “Separate Kernels Kill the Vibe.” [source]
The No Bubbles project measured this case with Llama-3.2-1B. Its vLLM and SGLang baselines split one forward pass into roughly one hundred kernels and used at most half of the H100’s memory bandwidth. The authors replaced that sequence with a single megakernel and reported more than 1.5 times the performance of the faster baseline.
In No Bubbles, the megakernel replaces the coordination previously supplied by separate kernel launches. An on-GPU interpreter assigns instructions to streaming multiprocessors, shared memory is paged between those instructions, and explicit counters track their dependencies. This also lets the program begin loading weights for upcoming work before the current instruction has fully finished.
No Bubbles sits in a wider group of megakernel projects. Mirage Persistent Kernel compiles tensor programs into SM-level task graphs executed inside a single multi-GPU megakernel. Event Tensor extends megakernel compilation to dynamic shapes and data-dependent work. Ada-MK moves scheduling decisions to compile time for fixed deployment configurations.
Figure 5. Separate kernels leave pipeline bubbles between tasks. MPK places both tasks inside one megakernel, allowing data movement and compute to overlap across them. [source]
Nothing was wrong with the individual kernels. The problem was that each boundary exposed too little of the forward pass to schedule it well.
5. Bespoke code may be getting cheaper
Figure 6. Models can now improve parts of their own execution stack. [source]
The cycles above depended on an expensive middle step: someone had to write and test the specialized path before its assumptions could become reusable. Coding models can now take on parts of that work.Manjunath Kudlur makes a closely related argument about deployment IRs in The Imminent Obsolescence of ONNX, TFLite, and the Ilk, focusing on how transformer workloads and coding agents weaken the economic case for universal intermediate layers.
Google says its first-party models now process more than 10 billion tokens per minute through direct customer API use. One percent of that annual volume is more than 52 trillion tokens. At a hypothetical marginal serving cost of one dollar per million tokens, a one-percent efficiency improvement would save more than 52 million dollars a year. Even at ten cents, it would save more than 5 million dollars. Providers do not disclose their blended serving cost, but Google also reported reducing Gemini’s serving unit cost by 78 percent during 2025.
After deploying GPT-5.6 Sol, OpenAI reported using the model to improve the system that serves it. Production GPU-kernel changes lowered serving cost by 20 percent, while changes to speculative decoding improved token-generation efficiency by more than 15 percent. The kernel diffs and evaluation harness are not public, so this is a reported production result rather than a reproducible technical case.
The Kimi K3 technical report shows what one version of this workflow looks like. Its kernel tasks range from individual operators to fused megakernels. Each task begins with a PyTorch reference implementation. Code that exceeds a numerical error threshold receives no reward, and correct code is scored by performance against an expert implementation and the hardware limit.
These are early examples, not a general cost curve. But they show coding models taking on parts of the fixed engineering work behind specialized execution.
6. The next workload may not fit
The history above can be read as a sequence of workload assumptions reaching their limits:
| Case | Earlier default | What changed |
|---|---|---|
| TF Lite → bespoke runtime | A static graph invocation described the complete inference workload. | The runtime owned the autoregressive loop, persistent state, memory plan, and kernels. |
| General ML deployment → llama.cpp | A general ML deployment stack was the reusable way to execute a model. | A transformer-specific inference path became reusable infrastructure. |
| Model runtime → vLLM + SGLang | Each model invocation was scheduled as an independent unit. | The serving framework owned cache allocation, request batching, and prefix reuse across invocations. |
| Operator kernels → No Bubbles + MPK | A forward pass ran as a sequence of roughly operator-sized kernels. | A persistent megakernel scheduled, pipelined, and synchronized work across operators. |
| Human-written → model-assisted specialized code | Writing, testing, profiling, and revising specialized execution code required human engineering time. | GPT-5.6 Sol and Kimi K3 show coding models taking on more of that work. |
Frameworks still reduce implementation, portability, and maintenance risk. They have also become the safe organizational default. This creates an uneven view of risk.
That asymmetry showed up recently when SpaceX/xAI said it was building a C training stack for a fixed GPU cluster and claimed a potential order-of-magnitude improvement over JAX. Much of the reaction treated it as reinventing the wheel or evidence that JAX had been underoptimized. An earlier post from a former xAI JAX/compiler lead describes the underlying constraint: JAX’s upstream abstraction did not expose enough control for the sophisticated GPU scenarios they were trying to optimize. The public criticism focused on the risk of replacing JAX. The risk that its boundary no longer fit the workload received much less attention.Some of the skepticism was probably about the controversial CEO, but in my experience, going against a framework has always been challenged.
The economics can justify what would otherwise look like reinventing the wheel. The IEA reports that data-center investment pushed capital spending by five large technology companies above $400 billion in 2025. At modern token volumes, even a modest efficiency gain can pay for substantial engineering work.
Reinventing the wheel is not always wasted work. It can reveal the state, memory behavior, and scheduling boundaries hidden by an existing framework. The result may remain bespoke, or it may become the foundation for the next framework.