Multimodal models

TL;DR

• Multimodal models accept multiple input types (image, audio, video) alongside text. The most impactful category today is vision-language models that combine images with text.
• A common architecture uses a vision encoder (ViT or SigLIP) to convert an image into patch embeddings, a projection layer to map those embeddings into the LLM's token space, then a transformer decoder for generation. Other systems use cross-attention or separate towers.
• CLIP (OpenAI, 2021) was an influential demonstration of contrastive learning for aligning image and text representations in a shared embedding space.
• Images are expensive: a single high-resolution image costs 85 to 1,600 tokens depending on the API and resolution setting. Token budgets must account for this.
• GPT-4o, Claude 3.7 Sonnet, and Gemini 2.0 are examples of production multimodal models with different reported strengths in OCR, spatial reasoning, and long-context understanding. Compare current versions on the target task rather than assuming a fixed ranking.
• Multimodal models hallucinate on fine-grained visual details: object counting, spatial relationships, and small text OCR are all weaker than they appear in demos.

30-Second Explanation

Mental model: modality-specific encoders turn images, audio, or video into representations that a shared model can relate to text and other modalities. This avoids some lossy handoffs, but precision tasks still need task-specific measurement and sometimes specialized detectors, OCR, or audio models.

The problem it solves

Most of the world's information isn't text. Enterprise documents are PDFs full of charts and tables. Product catalogs are images. Medical records include scans. Accessibility tooling needs to describe images to screen readers. A model that only processes text can't work with any of these directly.

Before multimodal models, a common pipeline was: run a specialized vision model (object detection, OCR) to produce text, then pass that text to an LLM. Information can degrade at the handoff. The OCR may misread a number in a chart, the object detector may miss context, and the LLM may reason confidently from corrupted input.

Serial pipelines can spend significant engineering time on handoff failures. For example, a vision model might output "revenue: $2.3M" when the chart shows $23M, and the downstream LLM may generate a confident but wrong financial summary.

The fundamental improvement: multimodal models see the image directly. No lossy intermediate extraction. The model reads the chart, recognizes the axes, and extracts the number from pixels. One model, one step, fewer failure modes.

What is it?

A multimodal model is a neural network that can process and reason across multiple data types (modalities) within a single architecture. The most common combination today is vision plus language: the model accepts images and text as input and generates text as output.

Think of it like a bilingual person who can read both English and Japanese. A text-only model is monolingual. It can only process information in one "language" (text tokens). A multimodal model is bilingual: it can read images (visual tokens) and text (language tokens) and reason about both together, translating freely between them.

The key insight is that images and text can be represented in the same mathematical space. Once you convert an image into a sequence of vectors that live alongside text token embeddings, the transformer's attention mechanism handles the rest. It attends to both visual and textual tokens, learning which parts of the image are relevant to the text query.

How it works

Vision encoders: turning images into tokens

A vision encoder converts an image into a sequence of embedding vectors, analogous to how a tokenizer converts text into token embeddings. The dominant architecture is the Vision Transformer (ViT), introduced by Dosovitskiy et al. in 2020.

ViT works by splitting the image into fixed-size patches (typically 14x14 or 16x16 pixels). Each patch is flattened and linearly projected into an embedding vector. A 224x224 image with 14x14 patches produces 256 patch tokens. A 512x512 image produces 1,024+ patch tokens.

SigLIP (Google, 2023) is a later alternative used in some vision-language systems. It replaces CLIP's softmax-based contrastive loss with a sigmoid loss and changes the normalization and batching trade-offs; whether it scales better depends on the training setup.

The choice of vision encoder is only one part of the quality picture. ViT-L/14 and SigLIP-SO400M can behave differently across tasks, while the projection layer, LLM backbone, resolution, and training data also matter. Measure the complete stack.

The projection layer: bridging two worlds

The vision encoder produces embeddings in its own dimensional space (e.g., 1,024 dimensions for ViT-L). The LLM expects embeddings in its space (e.g., 4,096 dimensions for Llama). The projection layer bridges this gap.

The simplest approach is a linear projection: a learned matrix that maps from vision dimensions to LLM dimensions. LLaVA (Liu et al., 2023) proved that even a simple two-layer MLP works remarkably well as a projection layer. More complex cross-attention projectors (like Flamingo's Perceiver Resampler) can compress the visual token count but add architectural complexity.

# Simplified projection layer (LLaVA-style)
class VisionProjection(nn.Module):
    def __init__(self, vision_dim=1024, llm_dim=4096):
        super().__init__()
        self.proj = nn.Sequential(
            nn.Linear(vision_dim, llm_dim),
            nn.GELU(),
            nn.Linear(llm_dim, llm_dim),
        )

    def forward(self, vision_embeddings):
        # vision_embeddings: [batch, num_patches, vision_dim]
        # output: [batch, num_patches, llm_dim]
        return self.proj(vision_embeddings)

After projection, visual tokens are concatenated with text tokens to form a single sequence. The LLM processes this unified sequence with standard self-attention. No separate cross-attention module is needed in the simplest architectures.

CLIP and contrastive learning

CLIP (Contrastive Language-Image Pre-training, OpenAI, 2021) is an influential architecture for vision-language representation learning. It trains an image encoder and a text encoder simultaneously so that matching image-text pairs produce similar embeddings.

The training process uses contrastive learning on 400M image-text pairs scraped from the internet. For each batch, CLIP maximizes the cosine similarity between matching pairs (image of a dog, text "a photo of a dog") and minimizes it between non-matching pairs (image of a dog, text "a photo of a car").

After training, CLIP's shared embedding space can support zero-shot classification: encode an image, encode candidate text labels, and pick the label with the highest cosine similarity. No task-specific training is needed for that basic procedure, although task-specific calibration can help. This pattern supports image search, content moderation, and vision encoders used in systems such as GPT-4V and LLaVA.

Fusion strategies: early, late, and cross-attention

How and when visual and textual information combine defines the model's architecture family.

Early fusion is a common architecture for generative vision LLMs. Once visual tokens are projected into the LLM's space, self-attention can handle cross-modal reasoning without a separate cross-attention module. The cost is a longer sequence and more compute; cross-modal quality and serving efficiency depend on the model and task.

Multimodal API interaction pattern

In production, multimodal models are accessed through APIs that accept images alongside text. The interaction follows a specific pattern: the client encodes the image (typically as base64 or a URL), sends it with the text prompt, and the model processes both modalities together.