What are multimodal AI agents? (And where they still misread)
The work that stayed manual is mostly work that arrives as a picture: a photographed invoice, a recorded sales call, a screen a rep has to click through. Multimodal AI agents read those inputs directly and act on them, which moves whole queues off human desks. Accuracy on messy real-world images is the constraint.
What makes an agent multimodal
A text-only agent needs a person to type in what happened. A multimodal agent takes the raw artifact instead: the photo of an invoice, the recording of a sales call, the screenshot of a billing portal, the scanned contract. It reasons over those inputs together with text, then acts on them, filing a record, drafting a reply, or clicking through a system that offers no integration.
Gartner put multimodal at 1% of generative AI solutions in 2023 and projects 40% by 2027. The operator's read on that: the input side of automation stops being a data-entry problem.
What it changes
Three queues open up first. Documents that arrive as pictures rather than structured files, which covers most supplier invoices and signed paperwork. Recorded conversations, where an agent scores a call nobody had time to listen to. And screen work inside systems that were never built to connect to anything.
Where it still breaks
Character accuracy on clean typewritten text runs past 99%. Accuracy on messy real-world images is nowhere near that. On ReceiptBench, a set of 10,656 real receipt photos, GPT-5 scored 0.71 and Gemini 3 Pro 0.74 on field extraction. On a 400-case benchmark of blurred and partly obscured IDs, receipts and prescriptions, GPT-4o answered without hallucinating roughly 30% of the time. Vision models also read a screen well and locate things on it badly, which is why screen-driving agents still misclick.
Point multimodal agents at high-volume work where a wrong field is cheap to catch, and keep a human approving anything that moves money.
Last updated: Sep 5, 2026