AI & ML Quick Path

Part 6: Building AI applications

Multimodal AI apps: images, documents and voice

Learning goal. Send images and documents to vision-capable models, build image-and-text search, and add speech input and output to an application.

In plain words

Multimodal models handle more than one kind of input or output: text with images, documents, audio or video. A vision-language model reads a photo or scanned page together with your instructions and answers in text, for example "extract the invoice number, date and total as JSON". Document AI combines text recognition (OCR) with layout understanding to read forms, invoices and receipts. Image-and-text search uses a shared embedding space (models trained like CLIP) so a text query can find matching photos. Voice apps chain speech-to-text, an LLM and text-to-speech, or use models that handle audio directly. The same rules apply as for text: validate structured output, keep a person in the loop for important decisions, and treat images and audio as personal data when they show or record people.

Diagram Many kinds of input, one application
  • InputTextyour instructions and questions
  • InputImages and photosread together with the text
  • InputDocumentsscanned pages, forms, invoices
  • InputAudio and videospeech or recordings
ModelMultimodal appone model, one set of instructions

Multimodal models take more than one kind of input or output.

Diagram Document AI
  1. InputScanned pageinvoice, form or receipt
  2. ProcessOCRreads the text on the page
  3. ProcessLayout understandingfinds which text is a field, a table or a total
  4. OutputStructured fieldsinvoice number, date, total

OCR means optical character recognition: reading text from an image.

Diagram Image-and-text search
  1. InputText queryfor example: damaged front bumper
  2. ModelShared embedding spaceimages and text turned into comparable numbers
  3. OutputClosest matchesphotos that sit near the query

Models trained like CLIP place matching images and text close together.

Diagram Two ways to build a voice app

Chained pipeline

  1. InputSpeech-to-text
  2. ModelLLM
  3. OutputText-to-speech

Audio-capable model

  1. InputAudio goes into the model
  2. ModelModel answers in audio

Real-world example

An insurer lets customers upload photos of car damage. A vision model drafts a description of visible damage and a claim category for an assessor, who confirms or corrects it. Image search finds similar past claims to keep decisions consistent.

Diagram Car damage claim with a person in the loop
  1. InputCustomer photosuploaded with consent
  2. ModelVision modeldrafts damage description and claim category
  3. ProcessAssessorconfirms or corrects the draft
  4. OutputDecisionconsistent, with similar past claims found by image search

How it works

  1. Collect inputs (photos, PDFs, audio) with consent; remove what you do not need.
  2. Send the file with clear instructions and a JSON output format.
  3. Validate fields (dates parse, totals are numbers, categories are allowed).
  4. Route low-confidence or high-value cases to a person.
  5. Evaluate per field on a labelled test set.
Diagram Reading a document safely
  1. InputCollect with consentremove what you do not need
  2. ProcessSend with instructionsask for a JSON output format
  3. ProcessValidate fieldsdates parse, totals are numbers, categories allowed
  4. OutputRoute or acceptlow-confidence or high-value goes to a person
  5. OutputEvaluate per fieldon a labelled test set

Repeat for every new upload

Low-confidence or high-value cases go to a person.

Python example

Sending an image to a vision-capable model through an OpenAI-compatible API (requires openai and a model that accepts images):

Python — read_invoice.py (conceptual)
import base64, os
from openai import OpenAI
client = OpenAI()   # or base_url=... for a local vision model
image_b64 = base64.b64encode(open("invoice.png", "rb").read()).decode()
reply = client.chat.completions.create(
    model=os.environ["VISION_MODEL"], temperature=0, response_format={"type": "json_object"},
    messages=[{"role": "user", "content": [
        {"type": "text", "text": 'Extract JSON with "invoice_number", "date" (YYYY-MM-DD) and "total". Use null if missing.'},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}}]}])
print(reply.choices[0].message.content)     # validate before use, as in the LLM API lesson

.NET (C#) example

Microsoft.Extensions.AI sends images as DataContent inside a normal chat message, with typed output:

C# — Program.cs
using Microsoft.Extensions.AI;

byte[] image = await File.ReadAllBytesAsync("invoice.png");
var message = new ChatMessage(ChatRole.User,
[
    new TextContent("Extract the invoice number, date (YYYY-MM-DD) and total. Use null if missing."),
    new DataContent(image, "image/png"),
]);
ChatResponse<Invoice> response = await chat.GetResponseAsync<Invoice>([message], new ChatOptions { Temperature = 0 });

if (response.TryGetResult(out var invoice) && invoice is not null && DateOnly.TryParse(invoice.Date, out _) && invoice.Total >= 0)
    Console.WriteLine($"{invoice.InvoiceNumber}: {invoice.Total} on {invoice.Date}");
else
    Console.WriteLine("Could not read the invoice reliably; send to a person.");

public record Invoice(string? InvoiceNumber, string? Date, decimal Total);

For voice, combine speech-to-text (for example Whisper.net, see Speech) with the chat client and a text-to-speech service.

Applied Maths - Statistics

Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.

Word error rate (how good is speech-to-text)

Where it is used: scoring a transcript from a voice or document-reading model against the correct text.

Word error rate

WER = (substitutions + deletions + insertions) ÷ words in the correct text

Substitution = wrong word. Deletion = missed word. Insertion = extra word.

In plain words: count every word the model got wrong, then divide by the length of the correct text. Lower is better.

Example 1: a short sentence. The correct text has 10 words. The model gets 1 wrong word and misses 1 word, with no extra words. WER = (1 + 1 + 0) ÷ 10 = 0.20, or 20%.

Example 2: noisy audio. The correct text has 8 words. The model misses 1 and adds 2. WER = (0 + 1 + 2) ÷ 8 = 0.375, or 37.5%.

Audio and image size (what you send to the model)

Where it is used: estimating upload size and why large inputs are shrunk or compressed first.

Raw size

audio bytes = samples per second × seconds × bytes per sample
image pixels = width × height

Raw means before compression.

In plain words: size grows with time for audio and with width times height for images.

Example 1: one minute of speech. 16,000 samples per second × 60 seconds × 2 bytes = 1,920,000 bytes, about 1.9 MB.

Example 2: shrinking a photo. A 4000 × 3000 photo has 12,000,000 pixels. Resized to 1000 × 750 it has 750,000 pixels. 12,000,000 ÷ 750,000 = 16, so 16 times fewer pixels to process.

Maths behind it

  • Shared embedding spaces and contrastive learning
  • Convolutional and transformer image encoders
  • Optical character recognition
  • Spectrograms (for audio)

Common mistakes

  • Trusting extracted numbers without validation (totals, dates, account numbers).
  • Sending photos that show people or documents with personal data without a reason and consent.
  • Testing on clean scans only; real uploads are blurry, rotated and partial.
Diagram Test set: clean versus real uploads

Clean scans only

  • Looks accurate in testing
  • Misses blur, rotation, partial pages

Realistic uploads

  • Blurry, rotated and partial images included
  • Field accuracy reflects real use

Quick check

  1. What does a vision-language model do?

    Show answer

    It reads images together with text instructions and answers in text, such as extracting fields from a document.

  2. Why validate fields extracted from an invoice image?

    Show answer

    The model can misread or invent values; validation catches impossible dates, totals and formats before they enter your systems.

  3. How can a text query find matching photos?

    Show answer

    By embedding both into a shared space where matching images and descriptions are close together.

Completion

Evidence of completion. An app that accepts a photo or scanned document and returns validated structured fields, tested on 20 examples with field accuracy reported.

Saved in this browser only. See all progress.