Vision Fallback

The Vision Fallback extension ensures that every image sent to the agent is properly analyzed — even when the primary model can't process images directly. It intercepts image content in messages and replaces it with text descriptions, using a configured vision-capable model.

Why It Matters

Many powerful coding models (reasoning-focused, code-specialized) don't support image input. Without vision fallback, these models would see [image omitted] and lose critical context from screenshots, diagrams, mockups, and error photos.

The Vision Fallback extension solves this transparently — the model never knows it can't process images. It receives rich text descriptions instead.

How It Works

The Fallback Chain

Image arrives in message
        │
        ▼
1. Main model supports images?
   YES → use main model directly
   NO  → go to step 2
        │
        ▼
2. Admin-configured default vision model?
   YES → use it to describe images
   NO  → go to step 3
        │
        ▼
3. Any vision model with auth configured?
   YES → pick best available (Anthropic > OpenAI > Google > OpenRouter)
   NO  → remove images with explanatory note

Interception Points

The extension hooks into two event points to ensure images are always described:

Caching

Image descriptions are cached with a 500-entry LRU cache, keyed by content hash. If the same image appears multiple times (e.g., in a multi-turn conversation), it's only analyzed once.

Example

User uploads a screenshot and asks a non-vision model about it:

User: "What's wrong with this error message?" [screenshot.png]

[Vision Fallback] Analyzing 1 image(s) before persisting user message...
[Vision Fallback] Using fallback vision model: anthropic/claude-3-5-sonnet-20241022

→ Screenshot described as:
  "A terminal window showing a TypeScript compilation error at line 42:
   'Property 'map' does not exist on type 'string'.'
   The file is src/utils/format.ts, and the variable 'result' is typed as string
   but .map() is being called on it."

Model (non-vision): "The error is on line 42 of format.ts — you're calling
  .map() on a variable typed as string. You need to make sure 'result' is
  an array before calling .map()."

Error Handling

If image analysis fails:

1. The primary model is tried first
2. If it fails, a fallback vision model is tried
3. If all fail, images are replaced with: [Image analysis from ...: <error message>]
4. A warning notification is shown in the UI

Configuration

The extension uses the default vision provider/model configured in Spectral settings:

// Settings → Default Vision Provider
// Choose which model handles vision fallback
settings.setDefaultVisionProvider("anthropic");
settings.setDefaultVisionModel("claude-3-5-sonnet-20241022");

If no default is set, it auto-discovers any configured vision model — preferring Anthropic, then OpenAI, then Google, then OpenRouter.

When It Activates

The extension activates when:

• The main model doesn't list image in its supported inputs
• A user message contains image content blocks
• A context is being built that includes image messages

It does not activate when:

• The main model already supports images natively
• There are no authenticated vision models available (images are simply omitted with a note)

Next Steps

• Wizard Integration
• SEO Analysis