Serving Multimodal LLMs — The New Challenges Image Input Creates

• Introduction: One Image Changes Serving
• What Differs From Text-Only
  • The Vision Encoder as an Added Stage
  • Variable Visual Token Count
  • Prefill Cost Spike
• Image Preprocessing and Caching
• Pipelining the Vision Encoder and LLM
• The Difficulty of Multimodal KV Cache and Batching
  • KV Cache Pressure
  • Batching Imbalance
• Latency Decomposition: Encoding Is Inside TTFT
• Throughput Optimization
• Cost
• Autoscaling and Load Characteristics
• Monitoring: What to Watch
• Concurrent Multi-Image and Long Context
• Robustness and Fallback
• Deployment Topology
• Framework Multimodal Support
• Worked Example: Tracing the Latency Decomposition
• Comparison: Text-Only vs Multimodal Serving
• Operational Pitfalls and Checklist
• Conclusion
• References

Introduction: One Image Changes Serving

Text LLM serving has standardized considerably over the past few years. Continuous (in-flight) batching keeps requests filling the pipeline, paged KV cache manages memory like virtual memory, and exploiting the fact that the decode stage is memory-bound, quantization and KV optimization push throughput up. Frameworks like vLLM, TensorRT-LLM, and SGLang implement this pattern well.

But the moment an image enters the input, the story changes. A multimodal LLM adds a vision-encoder stage in front of a pipeline that used to take only text tokens, the visual token count varies wildly per request, and prefill cost spikes compared to text. Assumptions that worked well in text-only serving start to wobble.

This article lays out how multimodal LLM serving differs from text-only serving and the concrete challenges and responses that difference creates. We cover the added vision-encoder stage, variable visual tokens, prefill cost, the difficulty of KV cache and batching, latency decomposition, throughput optimization, cost, and operational pitfalls and a checklist.

What Differs From Text-Only

The Vision Encoder as an Added Stage

A text-only LLM flows: tokenize, prefill, decode. Multimodal adds image preprocessing and vision encoding in front.

Text-only vs multimodal serving flow

[text-only]
  text -> tokenize -> prefill -> decode (generate tokens)

[multimodal]
  image -> preprocess (resize/normalize) -> vision encoder (ViT) ->
           projector (adapter) -> visual tokens
  text -> tokenize -> text tokens
  [visual tokens + text tokens] -> prefill -> decode

The vision encoder is heavy. A ViT forward pass consumes extra GPU compute, and the bigger and more numerous the images, the longer this stage. This added stage goes straight into time-to-first-token (TTFT).

Variable Visual Token Count

Input length is variable in text-only serving too, but multimodal varies more. The same "one image" can be 256 visual tokens or 4000 depending on resolution. This is even more true for models that support arbitrary resolution (dynamic resolution).

Variation in visual token count (concept)

request A: 1 small thumbnail      -> visual tokens ~256
request B: 1 high-res document    -> visual tokens ~3000
request C: 4 images               -> visual tokens ~4000+

Problem: sequence length varies widely per request
         making batch memory/compute hard to predict, causing imbalance

This variability complicates batching and memory planning. Mixing requests with vastly different token counts in one batch easily wastes padding or overruns memory.

Prefill Cost Spike

Many visual tokens means a long prefill sequence. Attention is quadratic in sequence length, so prefill compute spikes. Where text-only had short prefill and long decode, multimodal makes prefill itself a heavy task.

Prefill cost comparison (concept)

[text-only] 200-token prompt
  prefill: process 200 tokens once (relatively light)

[multimodal] text 50 + visual 3000 = 3050 tokens
  prefill: process 3050 tokens, attention O(L^2) term spikes
  -> most of TTFT comes from prefill + vision encoding

Image Preprocessing and Caching

Image preprocessing (decode, resize, normalize) and vision encoding are expensive but often reusable. When the same image appears across requests (multi-turn conversation, repeated queries on one document), you can cache and reuse the vision-encoding result.

Image/visual-token caching (concept)

request arrives -> compute image hash
  cache hit?  -> reuse cached visual tokens (skip vision encoder)
  cache miss? -> preprocess + vision encode -> store result in cache

Effect: cuts vision-encoding cost in multi-turn/repeat queries, shortens TTFT
Caution: cache key (include resolution/preprocessing params), manage memory limit

Preprocessing itself can be a CPU bottleneck. Offload large image decode/resize to GPU or separate workers, and use an asynchronous pipeline to reduce GPU idle time.

Pipelining the Vision Encoder and LLM

The vision encoder and the LLM have different compute characteristics. Running them serially leaves one idle while the other runs. Overlapping them via pipelining raises throughput.

Serial vs pipeline (concept)

[serial]
  [vision encode A][LLM prefill A][vision encode B][LLM prefill B]...
  -> idle between stages

[pipeline]
  vision encode:  [A][B][C]...
  LLM prefill:       [A][B][C]...  (start prefill as soon as A's encoding finishes)
  -> overlapping the two stages raises GPU utilization

In practice, a disaggregated setup is common: separate the vision encoder into its own stage (or GPU/server) to produce visual tokens ahead of time, while the LLM engine receives tokens and focuses on prefill/decode. This connects to the latency decomposition discussed below.

The Difficulty of Multimodal KV Cache and Batching

KV Cache Pressure

The KV cache stores key/value for every token in the sequence. With thousands of visual tokens, the KV cache grows accordingly. Paged KV cache reduces fragmentation but cannot prevent the absolute memory demand from rising.

Multimodal KV cache pressure (concept)

KV cache memory ~ (total tokens) x (layers) x (heads x dim) x 2(K,V)

adding 3000 visual tokens ->
  that request's KV cache is several to tens of times the text case
  -> fewer concurrent requests (smaller batch)

Batching Imbalance

Continuous batching fills requests at token granularity to raise throughput. But in multimodal, large sequence-length variation means one request with a long prefill can slow the whole batch's progress. Also, prefill (many visual tokens) and decode (one token at a time) have different compute characteristics that are awkward to mix in one batch.

Effective responses are chunked prefill (split a long prefill into pieces and interleave with decode) and prefill/decode disaggregation (separate prefill-only and decode-only instances). Because multimodal prefill is heavy, the benefit of this separation is larger than in text-only.

chunked prefill + disaggregation (concept)

split a long visual prefill into chunks:
  prefill_chunk_1 -> a few decode steps -> prefill_chunk_2 -> ...
  -> a long prefill does not starve decode requests

prefill/decode disaggregation:
  [Prefill instance] handles vision encoding + prefill
  [Decode instance]  receives KV and generates tokens
  -> optimize for each stage's characteristics

Latency Decomposition: Encoding Is Inside TTFT

Multimodal serving latency decomposes differently from text-only. The key point is that vision encoding falls inside TTFT (time to first token).

Latency decomposition (concept)

[text-only]
  TTFT = tokenize + prefill
  TPOT = decode time per token

[multimodal]
  TTFT = image preprocess + vision encoding + (visual+text) prefill
  TPOT = decode time per token (similar to text-only)

Implication: multimodal TTFT tends to grow a lot.
            Optimizing TTFT = reduce preprocessing/encoding/prefill all.

So to reduce TTFT, combine (1) accelerating/offloading image preprocessing, (2) caching/batching vision encoding, (3) shortening prefill length via token compression, and (4) spreading perceived latency with chunked prefill. Meanwhile TPOT (time per output token), since decode is memory-bound, responds to quantization, KV optimization, and speculative decoding much as in text-only.

Throughput Optimization

The levers for throughput largely overlap with text-only, plus multimodal specifics.

• Reduce visual tokens: cutting prefill via dynamic resolution caps and token compression directly raises throughput. Reducing tokens is the biggest lever.
• Batch vision encoding: gathering multiple images to run the vision encoder in a batch raises GPU utilization.
• prefill/decode disaggregation + chunked prefill: keep heavy prefill from blocking decode.
• Quantization: FP8/INT4 cuts memory/bandwidth to grow decode throughput and batch size.
• Speculative decoding: a draft model proposes candidates the main model verifies, giving roughly 2-3x on memory-bound decode.
• Framework choice: vLLM's strengths are broad model/hardware support and multimodal support; TensorRT-LLM is compile-based with roughly 15-30% higher throughput on H100; SGLang is strong at prefix cache reuse via RadixAttention.

Throughput lever priority (concept)

1) reduce token count (dynamic resolution, compression)  <- most direct
2) cache/batch vision encoding
3) prefill/decode disaggregation + chunked prefill
4) quantization (FP8/INT4)
5) speculative decoding

Cost

Cost ties directly to token count and GPU time. Because of visual tokens, multimodal makes the same "one request" pricier than text.

Cost drivers (concept)

request cost ~ (vision encoding GPU time) + (prefill FLOPs) + (decode time x output length)

visual tokens N_v increase -> prefill FLOPs and KV memory increase ->
  smaller batch -> higher per-request shared cost

savings levers: resolution policy, token compression, encoding caching, quantization to grow batch

So the core of cost management is "do not feed larger images than needed" and "cache reusable vision encoding." Every means of reducing tokens without losing accuracy is cost savings.

Autoscaling and Load Characteristics

Multimodal serving load is harder to predict than text-only. Even at the same requests per second (RPS), GPU load varies wildly with the size and count of images in a request. Autoscaling purely on RPS easily under- or over-provisions.

Choosing a load metric (concept)

[inappropriate] scale on RPS only
  -> at the same RPS, load varies greatly with visual token volume

[appropriate] base on visual token throughput (tokens/s) or GPU utilization
  -> better reflects actual compute load

Additionally: set queue wait time and TTFT p95 as SLOs and scale to them

Also, since vision encoding and decode loads are separated, in a prefill/decode disaggregated setup it is efficient to scale each pool independently. For image-heavy traffic, grow the prefill pool; for long-response-heavy traffic, grow the decode pool.

Monitoring: What to Watch

Multimodal serving has more metrics to observe than text-only, because stages are added and variability is high.

Key monitoring metrics (concept)

Latency series:
  - TTFT (includes preprocess + encoding + prefill) p50/p95/p99
  - TPOT (decode time per token)
  - per-stage decomposition: preprocess / vision encoding / prefill / decode

Throughput series:
  - requests/s, output tokens/s, visual tokens/s

Resource series:
  - GPU utilization (each when vision encoder vs LLM are separated)
  - KV cache occupancy, OOM occurrences
  - cache hit rate (vision encoding cache)

Quality/cost series:
  - average visual token count, cost per request

Decomposing TTFT by stage is especially important. When TTFT is high, the remedy differs entirely depending on whether the cause is preprocessing, vision encoding, or prefill.

Concurrent Multi-Image and Long Context

When a request carries several images, or images accumulate across a multi-turn conversation, context grows quickly. This creates two pressures at once: increased prefill compute and increased KV cache memory.

Multi-image/long-context pressure (concept)

a request with 4 images (~1000 tokens each) + text
  -> visual tokens ~4000, longer prefill

multi-turn: images and responses accumulate per turn
  -> KV cache grows with conversation length
  -> fewer concurrent sessions in long conversations

Responses:
  - cap image count, summarize/drop old image tokens
  - cache vision encoding to avoid re-encoding
  - reuse prefix cache (same prefix context)

When the same image recurs across turns, prefix cache reuse pays off greatly. Techniques that share a common prefix, like SGLang's RadixAttention, are especially effective in multimodal multi-turn.

Robustness and Fallback

A multimodal pipeline has many stages and thus many failure points. Corrupted images, unsupported formats, and oversized inputs can arrive. Robust serving must handle these gracefully.

Input validation and fallback (concept)

input received ->
  validate format/size (supported format? within max size?)
    fail -> return a clear error (block before entering the pipeline)
  corrupted image decode failure
    -> skip that image or return an error
  oversized resolution
    -> downscale to the cap, then proceed

Principle: do not let bad input reach the GPU stage and waste resources;
          validate/block early up front

GPUs are expensive, so filtering bad input before the preprocessing stage is advantageous for both cost and stability.

Deployment Topology

How you lay out multimodal serving directly affects throughput and cost. There are broadly three topologies.

Deployment topology (concept)

[single unified]
  one engine handles preprocess + vision encoding + prefill + decode
  pro: simple, easy to operate
  con: cannot scale stages independently, resource imbalance

[vision encoder separated]
  vision encoding service | LLM engine (prefill+decode)
  pro: independent vision/LLM scaling, easy vision-result caching
  con: token transfer overhead, more operational complexity

[fully disaggregated (prefill/decode split)]
  vision encoding | prefill instance | decode instance
  pro: optimal resource/scaling per stage, highest efficiency
  con: most complex, needs communication design (KV transfer, etc.)

It is common to start with single unified at small scale, then evolve to a separated vision encoder, and further to full disaggregation as traffic grows and the image share rises. The more you separate, the higher the efficiency but also the operational complexity, so introduce it gradually to match scale.

Framework Multimodal Support

Here is the big picture of how major inference frameworks support multimodal. Details change by version, so read it as a general trend.

Per-framework characteristics (concept)

vLLM:
  - broad model/hardware support, mature multimodal input support
  - paged KV cache, continuous batching, chunked prefill
  - a solid default for general serving

TensorRT-LLM:
  - compile-based optimization, high throughput on H100
  - multimodal needs a vision-encoder-integrated pipeline setup
  - accept build/ops cost when chasing peak performance

SGLang:
  - strong at prefix cache reuse via RadixAttention
  - large gains for multi-turn / common-prefix context
  - suited to multimodal multi-turn workloads

The selection criterion is workload character. If you must stand up many models fast, vLLM; if peak throughput for a single model matters, TensorRT-LLM; if multi-turn conversation is central, SGLang is a starting point. In practice, validating on your own traffic via benchmarks is the real answer.

Worked Example: Tracing the Latency Decomposition

To make the abstract discussion concrete, let us trace one multimodal request's latency stage by stage. The numbers are hypothetical relative values for illustration; reality varies greatly with model, hardware, and configuration.

Multimodal request latency decomposition (concept, hypothetical relative values)

request: 1 high-res image (visual tokens ~2000) + 50 text tokens
output: generate 200 tokens

per-stage contribution:
  image preprocess     : small
  vision encoding       : medium (grows with image size)
  prefill (2050 tokens) : large  (attention O(L^2) term)
  ----- TTFT ends here -----
  decode (200 tokens)   : per-token time x 200

Observations:
  - TTFT is a large share of total latency (encoding + long prefill)
  - decode behaves similarly to text-only

The lesson of this decomposition is clear. To reduce the user-perceived first-response latency (TTFT) in multimodal, decode optimization alone — the main focus in text-only — is not enough. Reducing vision encoding and prefill is key.

Effect of applying TTFT-shortening levers (concept)

baseline: large TTFT from encoding + long prefill

lever 1) token compression: visual tokens 2000 -> 600
  -> prefill drops sharply -> TTFT shortens greatly
lever 2) vision encoding cache hit
  -> encoding stage near 0 -> further TTFT reduction
lever 3) chunked prefill
  -> long prefill stops blocking other requests -> improves overall p95

Using the three levers together improves TTFT and throughput at once. The key is to always view the three directions together: reduce visual tokens, cache reusable encoding, and spread heavy prefill.

Comparison: Text-Only vs Multimodal Serving

Operational Pitfalls and Checklist

Pitfalls:

• Unbounded resolution input: high-res images explode tokens, spiking TTFT and cost. Dynamic resolution caps are essential.
• Overlooking the vision encoder bottleneck: optimizing only the LLM while neglecting the vision encoder/preprocessing makes that the bottleneck.
• Batch imbalance: mixing requests with large length variance lets a long prefill starve short requests. Mitigate with chunked prefill/disaggregation.
• KV memory overrun (OOM): memory plans that overlook visual tokens cause OOM. Budget against the maximum visual tokens.
• Cache key errors: omitting preprocessing params (resolution/normalization) from the cache key reuses wrong visual tokens.
• Preprocessing CPU bottleneck: serialized image decode/resize on CPU starves the GPU. Offload/async it.

Checklist:

Multimodal serving checklist

[ ] Is there a dynamic resolution cap and token compression policy?
[ ] Is vision-encoding result caching (key includes preprocessing params) in place?
[ ] Are the vision encoder and LLM pipelined/disaggregated?
[ ] Is chunked prefill or prefill/decode disaggregation applied?
[ ] Is KV memory budgeted against maximum visual tokens?
[ ] Are TTFT/TPOT/throughput monitored separately?
[ ] Is the preprocessing CPU bottleneck offloaded/async?
[ ] Is FP8/INT4 quantization used to secure batch size?

Conclusion

Multimodal LLM serving is an extension of text-only serving, but the difference image input creates is essential. The added vision-encoder stage, visual tokens that vary per request, and heavier prefill change the TTFT, memory, and cost structure.

The broad response is clear: reduce tokens (dynamic resolution/compression), cache/batch/pipeline encoding, disaggregate/chunk prefill, and accelerate decode with quantization, KV optimization, and speculative decoding. As multimodal support in frameworks like vLLM matures, these patterns are relatively easy to apply. In the end, the core question converges to one: how much can you reduce visual tokens and prefill cost while preserving accuracy?

References

• vLLM docs
• vLLM GitHub
• SGLang GitHub
• TensorRT-LLM GitHub
• Qwen2-VL (arXiv 2409.12191)
• FlashAttention (arXiv 2205.14135)
• Attention Is All You Need (arXiv 1706.03762)
• QwenLM GitHub
• Hugging Face docs