Vision and Multimodal: Image Understanding, OCR, and VLMs

• Vision Models Are Not One Category
• Conversational VLMs That Understand Images
• Small VLMs and On-Device
• Dedicated OCR Is a Different Object from a VLM
• Document Structuring Is Not Text Extraction
• Image Retrieval Has Its Own Embedding Models
• Prohibited Uses and Limitations Stated on the Cards
• Code Example
• The Order to Decide In
• Try It Yourself
• Series Navigation
• References

Model details were read directly from the Hugging Face pages on 2026-08-12. Model cards and licenses change, so check the original again before you use anything.

Vision Models Are Not One Category

Search for open models that handle images and very different things come back in one list. In practice you need at least four buckets before a choice is possible: VLMs that converse about an image, dedicated OCR models that read characters precisely, models that reconstruct document structure, and embedding models that turn images into vectors for retrieval.

These four do not substitute for one another. Hand a receipt to a VLM and it will usually read it, but you have no way to confirm it did not skip the amount field. A dedicated OCR model reads precisely but never judges whether the receipt is approvable.

Conversational VLMs That Understand Images

The Qwen/Qwen2.5-VL-7B-Instruct card states it can produce structured outputs of the contents of invoices, forms, and tables, and that it can localize objects in an image by generating bounding boxes or points. In an automation pipeline that localization capability is decisive. A model that returns only values forces a human to compare against the original, while a model that returns coordinates can highlight the exact spot on a review screen.

The same card states that the YaRN extension technique has a significant impact on temporal and spatial localization tasks for inputs beyond 32,768 tokens and is therefore not recommended. In other words, do not demand long video and precise localization at the same time.

openbmb/MiniCPM-V-2_6 states it processes images of any aspect ratio up to 1.8 million pixels. Unlike models that resize to a fixed resolution, that means tall screenshots and wide diagrams are not cropped away, which matters for use cases like screenshot analysis.

Small VLMs and On-Device

HuggingFaceTB/SmolVLM-Instruct is Apache 2.0 with 2B parameters, and its evaluation table lists a minimum required GPU RAM of 5.02 GB. Having that number on the card at all is useful, because most VLM cards do not state memory requirements and you have to measure them yourself.

The openbmb/MiniCPM-V-2_6 card states it can be deployed on end-side devices such as iPad. If you are evaluating on-device vision, these two repositories are the starting point.

Dedicated OCR Is a Different Object from a VLM

stepfun-ai/GOT-OCR2_0 is apache-2.0 at 0.7B, and its card lists several OCR modes: plain text OCR, format-preserving OCR, fine-grained OCR that targets a region by box or color, and multi-crop OCR, with formatted results renderable to HTML.

The operationally important one is fine-grained OCR. For a fixed-form document you can read only the fields you need by coordinate instead of reading and parsing the whole page, and then a misread elsewhere never contaminates the result. No limitations section is stated on the card.

microsoft/Florence-2-large is mit at 0.77B and works by specifying the task with a prompt token. The card lists tokens such as <CAPTION>, <DETAILED_CAPTION>, <OD>, <OCR>, <OCR_WITH_REGION>, <DENSE_REGION_CAPTION>, <REGION_PROPOSAL>, and <CAPTION_TO_PHRASE_GROUNDING>. Its usage example states trust_remote_code=True is required, and the card itself notes this is a continued-pretraining version with 4k context where only 0.1B samples were used, so it might not be trained well.

Document Structuring Is Not Text Extraction

When you work with contracts or papers, what you need is not characters but structure. A table has to remain a table, a paragraph has to stay under its heading, and a formula has to stay a formula.

docling-project/SmolDocling-256M-preview is cdla-permissive-2.0 at 0.3B and states it uses an output format called DocTags. The capabilities the card lists are OCR, layout and localization with bounding boxes, code recognition, formula recognition, chart recognition, table recognition with headers, figure classification, caption correspondence, list grouping, and full-page conversion.

Two facts to confirm before using it: preview is in the name, and the card states that granite-docling-258M has been released as its successor and will now receive updates and support. Even entering through the ds4sd/SmolDocling-256M-preview address, the page displays docling-project/SmolDocling-256M-preview.

Image Retrieval Has Its Own Embedding Models

Finding images by text or finding similar images is a job for an embedding model rather than a VLM. google/siglip-so400m-patch14-384 is apache-2.0 at 0.9B and states it can be used for zero-shot image classification and image-text retrieval. It was pre-trained at 384x384 resolution.

One sentence on this card is worth noticing: the team releasing SigLIP did not write a model card for this model, so the Hugging Face team wrote it. What is stated there is therefore a third-party summary rather than a publisher-warranted specification, so cross-checking against the original paper is the safer route. No limitations section is stated.

Prohibited Uses and Limitations Stated on the Cards

Because vision models act directly on people and documents, their prohibited uses are often spelled out concretely.

The HuggingFaceTB/SmolVLM-Instruct card states the model is not intended for high-stakes scenarios, lists employment evaluation and critical automated decision-making as prohibited uses, and notes it does not support image generation. The openbmb/MiniCPM-V-2_6 card states the model cannot comprehend or express personal opinions or make value judgements, and that the developers will not be liable for problems arising from its use. google/gemma-3-4b-it and google/gemma-3-27b-it state it may fail to grasp subtle nuance, sarcasm, or figurative language and may generate incorrect or outdated factual statements.

License structures can be compound. openbmb/MiniCPM-V-2_6 states the code is Apache-2.0 while the model follows a separate MiniCPM Model License, with free commercial use after completing a registration questionnaire. Read the full license text yourself and put commercial use through legal review.

Code Example

# Example: send an image with a question and get a structured answer back
from transformers import AutoProcessor, AutoModelForImageTextToText

repo = "Qwen/Qwen2.5-VL-7B-Instruct"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype="auto", device_map="auto")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "invoice.png"},
            {"type": "text", "text": "Return the line items and amounts as JSON."},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_tensors="pt"
).to(model.device)
out = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(out[0], skip_special_tokens=True))

The Order to Decide In

1. Separate first whether you need understanding, extraction, structure reconstruction, or retrieval.
2. Check the real resolution and aspect ratio distribution of your inputs and whether the model crops them away.
3. Decide whether you need positional information. If so, keep only models that emit coordinates.
4. If several images go in at once, pick a model that states multi-image conversation.
5. Confirm your use case is not on the prohibited list.
6. Check for preview labels and successor notices to pick the one that will be maintained.

Try It Yourself

• Neural Network Architecture Explorer — look at what vision architectures actually capture.
• Neural Network Lab — experiment with how image feature extraction works.
• GPU VRAM Calculator for LLMs — a VLM has to account for image tokens too, so size the memory in advance.

Series Navigation

• Previous: Choosing Speech Models: Practical Criteria for STT and TTS
• Next: Choosing Code Models: Completion vs Chat, FIM, and Licenses

References

• Every value in the tables was read directly from that model page on Hugging Face on 2026-08-12. Anything absent from the page is written as not stated.
• The limitations sections of stepfun-ai/GOT-OCR2_0 and google/siglip-so400m-patch14-384 are not stated on their pages, so nothing is written here.
• Read the full license text yourself and put commercial use through legal review.