Multimodal Prompting

Intermediate

Show. Don't just tell.

01 The Concept

Modern AI doesn't just read — it sees. You can hand it screenshots, photos, diagrams, PDFs, spreadsheets, and handwritten notes. The model processes them as first-class inputs alongside your text.

But most people use multimodal like a parlor trick: 'What's in this image?' That wastes the capability. The real power is combining visual input with structured text prompts — giving the AI both the thing to analyze AND specific instructions for how to analyze it.

The rule: an image replaces a thousand words of description, but you still need the prompt to tell the model what to DO with what it sees.

02 Weak vs. Strong

EX 01UI screenshot review for real issues

You are a senior UX designer who has shipped 20+ B2B SaaS products.

I'm attaching a screenshot of our dashboard's main view. Review it against these specific criteria:

Visual hierarchy — is it clear what the user should look at first, second, third?
Information density — is any section overloaded or underused?
Accessibility — contrast issues, text too small, missing labels, color-only indicators?
Mobile risk — which elements will break on a 375px viewport?
Action clarity — can a new user find the primary action within 3 seconds?

For each issue:

Quote the specific element (e.g., 'the revenue chart in the top-right')
Severity: [CRITICAL] [WARNING] [SUGGESTION]
What to change and why — be specific enough for a developer to implement

Context: this is a project management dashboard for teams of 5–20. Our users are non-technical project managers. The primary action should be 'Create New Task.'

[attaches screenshot]

→ Why it works

Specific criteria prevent generic 'looks good' feedback. Element-level quoting makes issues actionable. User persona context tells the model who's judging the design.

EX 02Extracting structured data from a photographed document

I'm attaching a photo of a restaurant receipt. Extract the data into this exact JSON schema:

{
  "restaurant": "string",
  "date": "YYYY-MM-DD",
  "items": [
    {
      "name": "string (as printed, preserve original spelling)",
      "quantity": "number",
      "unit_price": "number (2 decimal places)",
      "total": "number (2 decimal places)"
    }
  ],
  "subtotal": "number",
  "tax": "number",
  "tax_rate_pct": "number or null if not shown",
  "tip": "number or null if not shown",
  "total": "number",
  "payment_method": "string or null if not shown",
  "confidence_notes": ["string — list anything unclear, smudged, or partially visible"]
}

Rules:

Output ONLY valid JSON, no markdown wrapping
If a value is smudged or unclear, use your best guess AND add a note in confidence_notes
If quantity isn't shown, assume 1
Use the printed total, don't recalculate

[attaches receipt photo]

→ Why it works

Typed schema eliminates format guessing. Null rules handle missing data gracefully. Confidence notes surface uncertainty instead of hiding it. Rules prevent common OCR-to-JSON failures.

03 Key Points

01Always pair images with specific analytical instructions — 'What's in this image?' wastes the capability

02For UI review: specify what to evaluate (layout, accessibility, hierarchy, mobile responsiveness)

03For documents: tell it the document type, what to extract, and the output schema

04Multiple images in one prompt enable comparison: 'Compare design A vs B against these 5 criteria'

05Describe what the model CAN'T see: business context, user personas, technical constraints

04 Model-Specific Notes

Claude processes images inline with text. Place analytical instructions BEFORE the image when possible — Claude attends to instructions more reliably when they precede the visual input.

05 For Your Role

Instead of describing what you see, just show it. But always tell the AI what to LOOK FOR — 'review this screenshot' is as vague as 'review this document.'

Next: Structured Outputs & JSON Schema Modes →