llama.cpp Comes to the Browser — The Ceiling a WebGPU Backend Measured Across 16 Devices

Introduction — Has Browser LLM Graduated From Demo?
LlamaWeb — What It Means to Bolt a WebGPU Backend Onto llama.cpp
The Real Constraint Isn't VRAM — It's the Tab
The Numbers — Against Existing Browser Frameworks
The Name of the Ceiling — subgroup matrix
The Gap to Native — 2.5x to 10x
Safety Isn't Free — The Bounds-Check Bill
Quantization Here Isn't a Speed Trick
Browser Variance — Firefox Got Dropped Entirely
For Practitioners — Feature Detection and Fallback
So, Should You Run an LLM in the Browser
Closing
References

Introduction — Has Browser LLM Graduated From Demo?

The idea of running an LLM in the browser has lived in demo territory for years. WebLLM showed Llama running inside a tab; Transformers.js ported the pipeline API to the web; and every time, the same slogan came along — "zero server cost, data never leaves the device." The slogan is true. The problem is that almost no public data answered the next question: so how slow is it really, how much does it eat, and where does it break? Most benchmarks came from a single author's laptop, and browser GPU performance doesn't generalize across hardware.

That's what makes Llamas on the Web, posted to arXiv on May 20, 2026, interesting. Written by Reese Levine, Tyler Sorensen, and others at UC Santa Cruz, the paper builds a WebGPU backend for llama.cpp (the paper calls it LlamaWeb) and measures it across 16 GPUs from 8 vendors, 10 models, and 4 weight formats. In the authors' words, it's "the largest cross-device, cross-model evaluation dataset collected within a single inference-engine framework," and the only WebGPU dataset that includes mobile devices.

This post lays out what that data says about the reality of browser inference in 2026. The conclusion up front — it's gotten better, but the ceiling is now very clearly visible. And that ceiling has a name.

LlamaWeb — What It Means to Bolt a WebGPU Backend Onto llama.cpp

Let's lay out the terrain first. WebGPU is an abstraction the browser layers on top of several native APIs — Metal on macOS/iOS, Vulkan on Linux and Android, DirectX on Windows. And each browser implements it differently. Chromium-based browsers use Dawn, Firefox uses wgpu, and Safari uses its own WebKit implementation. One important fact here — Dawn is written in C++, so it can also be used from native code outside the browser. That fact turns out to be decisive later on.

Existing browser inference frameworks each leaned on a different engine — Transformers.js on ONNX Runtime, WebLLM on MLC-LLM. LlamaWeb instead chose to bolt a WebGPU backend directly onto llama.cpp itself. Why that matters: it inherits llama.cpp's assets wholesale — especially the entire GGUF quantization format ecosystem.

One of the key pieces of this work is already merged into llama.cpp proper. PR #17031 ("ggml webgpu: faster matrix multiplication/matrix-vector multiplication", reeselevine, merged November 8, 2025) added two matrix-multiplication kernels. To carry the PR description over directly — one uses register tiling, the other uses "subgroup matrices" (a WebGPU feature that gives access to tensor cores, or optimized subgroup routines), and "subgroup matrices are currently experimental, and fall back to the register tiling approach on unsupported devices."

Keep that fallback sentence in mind. Everything else in this post is really about that one sentence.

The Real Constraint Isn't VRAM — It's the Tab

This is where people new to browser inference most often get it wrong. The constraint isn't GPU memory — it's the tab process.

The first problem the paper identifies in existing frameworks is memory inefficiency. LlamaWeb allocates memory statically at startup, making memory usage predictable regardless of prompt length or generation length. It even reserves the intermediate storage various kernels need before the model runs for the first time.

The more interesting part is model loading. The authors rewrote the loading path while adding WebGPU support to wllama (originally a library that ran llama.cpp CPU-only via WASM). Weights are stored in OPFS (Origin Private File System), and never materialized into the WebAssembly heap at all. The reason is in a paper footnote, and once you read it, it makes sense.

Because the WASM heap is grow-only, memory a tab has once allocated cannot be released for as long as that tab is alive.

In other words, even briefly touching 2GB during loading and then letting go doesn't give that 2GB back until the tab closes. So LlamaWeb uses llama.cpp's asynchronous loading interface to stream weights from OPFS to WebGPU buffers using just four 1MB buffers. The paper notes that "Safari in particular has tight memory limits, and this change let us serve larger, more capable models."

The resulting numbers: measured peak memory across Chrome and Safari on multiple device combinations, an average 29–33% reduction versus existing browser inference frameworks. By the geometric mean of normalized peak memory across four test configurations on f16 Llama3.2 1B, it came in 49% lower than WebLLM and 41% lower than Transformers.js. Broken down by configuration, the spread is wide — on Chrome on an Apple M4 Pro it was only a 13% improvement over WebLLM and 16% over Transformers.js, but on Safari on the same device that widens to 28% and 59%.

However, part of this large gap isn't LlamaWeb doing well — it's the competition leaking. The paper explicitly states that on Safari, Transformers.js's memory climbed to 10GB before the tab died, which looks like a memory leak. On the RTX 5080 Windows configuration, WebLLM climbed to roughly 10GB the same way. When reading benchmarks, don't erase sentences like this — a good chunk of "uses 61% less" is the other side's bug.

The Numbers — Against Existing Browser Frameworks

The performance comparison was run on four GPUs: NVIDIA RTX 5080, Apple M4 Pro, AMD RX 7900 XT, and Intel Arc B580. The key point is that every vendor is different.

On decode (token generation) throughput, LlamaWeb was about 54% higher than WebLLM and 69% higher than Transformers.js. In the paper abstract's phrasing, it's a "45–69% improvement in decode throughput across four GPUs."

But prefill (prompt processing) runs the other way.

Prefill performance, however, lags — LlamaWeb reaches only 49% of WebLLM's throughput and 79% of Transformers.js's throughput.

Less than half. The paper points to two causes. First, WebLLM uses TVM's sophisticated kernel fusion to cut the cost of repeatedly re-reading model weights for every small operation. Second, Transformers.js has an implementation that optimizes matrix multiplication using subgroups.

This reveals the character of browser LLMs. Decode is dominated by memory-bandwidth-bound matrix-vector multiplication; prefill is dominated by compute-bound matrix-matrix multiplication. LlamaWeb has built the former well, and for the latter it still lacks a tool. What tool? —

The Name of the Ceiling — subgroup matrix

In WebGPU, subgroups are a feature that lets threads within a workgroup exchange values with each other in a SIMT fashion. This has already shipped — enabled by default on both desktop and Android in Chrome 134 (Chrome Platform Status), with more being added bit by bit since. subgroup_id is slated for Chrome 144, subgroup_uniformity for Chrome 145, and Subgroup Size Control for Chrome 152. (As of today, stable is Chrome 151, and 152's scheduled stable date is August 25, 2026.)

But subgroup matrix is different. This is a feature that directly unlocks the GPU's fixed-size matrix-multiply unit — tensor cores, in NVIDIA terms — and LLM prefill wants exactly that unit. The paper's kernel library already includes an sg_mat kernel that uses it. The problem is this one sentence.

The subgroup matrix feature isn't yet available in stable browser releases, but it is available natively in some WebGPU implementations.

Checking confirms this. On Chrome Platform Status, the subgroup-related entries are exactly Subgroups (134, enabled by default), subgroup_id (144), subgroup_uniformity (145), and Subgroup Size Control (slated for 152) — there is no subgroup matrix entry at all. Google is at the stage of saying it will bring a cooperative matrix proposal to the standards group, and as with subgroups, a large part of the discussion is expected to be about portability.

So the authors take a detour. Remember the earlier fact that Dawn is a C++ library — running the same backend code natively through Dawn lets you turn subgroup matrix on. Measure it that way, and the picture flips.

In fact, using LlamaWeb's native-run prefill numbers, LlamaWeb outperforms the other frameworks' prefill numbers on every device except WebLLM on the Apple M4 Pro, showing a geometric-mean speedup of 88% over WebLLM and 205% over Transformers.js.

Same code, same device, different execution environment. 49% of WebLLM inside the browser, 188% of WebLLM outside it. Today's browser-LLM prefill ceiling isn't algorithms or kernel quality — it's one un-standardized hardware feature.

The paper's conclusion strikes exactly that tone — the authors write that "while subgroup matrix isn't yet in browsers, once it reaches general availability, LlamaWeb is well-positioned to deliver competitive prefill performance." The load-bearing word is "once," not "is." Don't read past that conditional clause.

The Gap to Native — 2.5x to 10x

So what happens if we leave the browser and compare against native backends? The paper runs native backends on the same devices using llama.cpp's llama-bench for comparison.

As Fig. 5(a) shows, the CUDA and Vulkan backends still outpace WebGPU by up to 10x on prefill and 2.5x on decode, across both weight formats. This appears to be due to better utilization of tensor cores and access to vendor-specific advanced features such as asynchronous memory loads.

The Apple side is similar — "the Metal backend always outperforms the WebGPU backend, by more than 2x on prefill and 50% on decode."

This same gap shows up in the PR #17031 benchmark we saw earlier. It's preliminary numbers the PR author measured on their own M3 (that is, not a vendor report but a single author's single device, and llama-bench is llama.cpp's native CLI benchmarking tool).

Llama-3.2-1B-Instruct, Apple M3, llama-bench (measured by the PR #17031 author, preliminary numbers)

                      pp512 (t/s)         tg128 (t/s)
  F16   WebGPU      1014.17 ± 9.38       28.71 ± 0.19
  F16   Metal       1368.47 ± 0.95       35.99 ± 0.78
  Q4_0  WebGPU       960.52 ± 6.05       41.76 ± 0.62
  Q4_0  Metal       1346.68 ± 1.21      103.92 ± 0.37

Look at Q4_0's decode — 41.76 versus 103.92. That's 2.5x. The exact same order-of-magnitude gap as the paper's numbers above reproduces independently here.

But the direction isn't only one way. This is the most surprising part of the paper — there really are cases where WebGPU beats vendor-specific backends.

On Apple, vs. Vulkan: against the Vulkan backend running through MoltenVK, WebGPU is up to 59% faster on prefill and 45% faster on decode.
On AMD: the native HIP backend always outperforms WebGPU, but the Vulkan backend trails WebGPU by 3x on prefill.
On Intel: the WebGPU backend with safety checks turned off is the fastest on f16 model prefill, beating the SYCL backend by 23%, and it also beats native SYCL by 10% on decode for the same model.

The paper interprets this as an effect of cross-device tuning — even though WebGPU's underlying runtime on Linux is Vulkan itself, there are devices where LlamaWeb's tuned matrix-multiply parameters beat the Vulkan-specific backend. The intuition that "native is always faster" is wrong. More precisely: a well-tuned portable implementation beats a poorly-tuned native one.

Safety Isn't Free — The Bounds-Check Bill

In the browser, the WebGPU compiler inserts safety checks into shaders, to prevent out-of-bounds buffer access and division by zero. This isn't negotiable — you can't strip it out of a model where any arbitrary webpage can throw a shader at your GPU.

The paper measured what that costs.

Comparing performance with safety checks on versus off, prefill slows down by an average of 14% for q4_k_m models and 23% for f16 models, while decode slows down by 5% and 1% respectively. This slowdown reached as much as 42% on prefill for the NVIDIA RTX 5080.

In other words, safety checks eat double digits on prefill and are close to free on decode. This fits the same overall picture — a check gets attached to every index inside a compute-intensive matrix-multiply loop. And look again at the earlier Intel case: WebGPU beat SYCL under the condition of "safety checks turned off." That condition isn't available in the browser.

This shows up directly on the paper's list of future work — removing unnecessary bounds checks through better static analysis, handling floating-point differences more robustly at the language level to guarantee model stability under low-precision execution, and adding new data types like u16 to WebGPU to enable more efficient memory loading and native hardware acceleration.

Quantization Here Isn't a Speed Trick

This is a part that surprises you more the more experience you have with local LLMs.

As it currently stands, llama.cpp quantization on WebGPU is a technique for reducing the memory requirement needed to run a given model, not a performance optimization.

The reason q4 is used on CPU or CUDA is memory, sure, but also to save bandwidth and gain speed. On WebGPU, that equation doesn't hold. The paper gives two reasons. First, the dequantization routine for general-purpose weight formats is complex and compute-heavy — unlike hardware-native formats such as nvfp4, which WebGPU doesn't support. Second, the current layout may not be optimized for performance across the diverse range of GPUs WebGPU targets.

The M3 table above backs this up — Q4_0's prefill (960) is actually slower than F16's prefill (1014). Even though the file size dropped to a third, from 2.30GiB to 729.75MiB. On decode, quantization wins at 41.76 versus 28.71 (bandwidth dominates there), but on prefill, the dequantization cost eats the gain.

The practical implication is simple. In the browser, quantization is a means of "making a model that didn't run, run," not a means of "making a model that already ran, run faster." And given that, as we saw in the earlier section, the browser's real constraint is tab memory, that alone is worth plenty.

Browser Variance — Firefox Got Dropped Entirely

Leaving this paragraph out of a post about portability would be dishonest.

Firefox does support WebGPU, but we observed significantly lower performance. For example, on Apple M3, a llama model with q4_k_m weights ran at about 1 token per second on Firefox versus about 52 tokens per second on Chrome, so we exclude Firefox results from our evaluation.

That's 50x. The paper writes that Chrome, and Dawn underneath it, "consistently provided the highest and most stable WebGPU performance across many platforms," so unless otherwise noted, every experiment ran on Chrome. On iOS, no other browser engine is available, so Safari is used.

The sentence "WebGPU is supported by all three major browsers" is true, but leaning on it in production will hurt you. Feature support and performance parity are different axes, and as of July 2026, the latter is nowhere close to being in place.

There's a landmine on the floating-point side too. Look at one paper footnote — during development, using f16 accumulation in the register-tiling kernel caused some models like Qwen2.5 to produce gibberish output on Apple M-series GPUs, so that kernel currently uses f32 accumulation instead. In the authors' words, "differences in floating-point precision are a well-known problem, and handling them within WebGPU and its applications is not a solved problem." The nasty part of this class of bug is that it shows up not as a crash, but as silent quality degradation.

For Practitioners — Feature Detection and Fallback

At the code level, all of this boils down to one thing. Don't assume a feature — ask for it.

const adapter = await navigator.gpu.requestAdapter();
if (!adapter) throw new Error('WebGPU 어댑터 없음');

// subgroups is enabled by default in Chrome 134+, but you still need
// to check that the adapter actually advertises it and request it explicitly.
const wanted = ['subgroups', 'shader-f16'].filter((f) => adapter.features.has(f));

const device = await adapter.requestDevice({ requiredFeatures: wanted });

// subgroup matrix is not in any stable browser as of July 2026.
// Don't write code that pretends it exists — treat the fallback path as the default path.
const hasSubgroups = device.features.has('subgroups');

LlamaWeb's design is exactly this shape — the sg_mat kernel exists, but it isn't the default path, and it falls back to register tiling on unsupported devices. To carry PR #17031's sentence over again, "on unsupported devices, the code falls back to the register tiling approach." In the browser, right now, that fallback is the path for every device.

One more thing. Kernel compilation incurs a one-time cost (roughly 1–5 seconds) during the model's first forward pass, after which it runs from a cached pipeline. Don't forget those few seconds when designing for first-token latency.

So, Should You Run an LLM in the Browser

Looking at the data, the decision criteria become fairly clear.

When it earns its cost

Data must not leave the device. This is nearly the only overwhelming reason — it's the one kind of requirement worth accepting 2.5x slower for.
The model is small. The models the paper actually measured range from 0.23GB to 3.11GB.
Decode-dominant workload — short prompt, long generation. This is where browser inference loses the least.
Deployment has to be a single URL. No install, no runtime, no server — that's still real value.

When it's not there yet

Prefill-dominant — patterns like RAG where you feed in an entire long document. This is precisely where the gap widens to as much as 10x, and the feature that would fix it isn't in the standard yet.
You need to support Firefox. 50x isn't a performance issue — it's a feature issue.
You're sensitive to precision. The paper already documents a case where f16 accumulation quietly wrecked output.
You can use a server GPU and have no privacy constraints. Then just run it on the server. Browser inference is a privacy and deployment technology, not a cost-saving one.

That last item matters most. "Zero server cost" is true, but in exchange you take on your users' battery, 2.5–10x the performance gap, and hardware variance across 8 vendors. That's not savings — it's a transfer.

Closing

To sum up the state of browser LLM inference in 2026 in one paragraph: the engineering has matured — static memory planning and OPFS streaming cut memory by 29–33%, and tuned portable kernels beat vendor-specific native backends on some devices. Yet on prefill, it still loses to native by up to 10x, and a large chunk of that reason is the absence of a standard feature that unlocks tensor cores. The proof is that running the same code outside the browser with that feature flips the ranking.

So the lesson of this story isn't "the browser is still slow." It's actually the opposite — the bottleneck now sits not in software but in the standardization timeline. Given that subgroups went through roughly a year of origin trial from the original proposal to being enabled by default in Chrome 134, it's clear what it means that cooperative matrix is still at the proposal stage. And the center of that discussion will be portability, not performance — covering 8 vendors' matrix units with one safe API is inherently hard.

Until then, deciding to run an LLM in the browser is a privacy decision, not a performance decision. Approach it with that frame, and this technology already earns its keep today. Approach it with a benchmark frame, and you'll be disappointed.