Ask HN: Local LLM agents on Jetson/RPi without a heavy runtime
I'd like to design agent flows on my Mac and ship a single tiny binary to Jetson/RPi. LLMs run on-device (no cloud). If you've tried this, what constraint hit first? Pointers to tools or write-ups you liked are welcome.
I don't have much experience but I played around with a USB Coral TPU. I did not try it with an LLM, but with used some image recognition stuff. Seemed OK, but wasn't as intense as an LLM would be. You'd probably have better luck with the Jetson you mentioned, or with one of Coral's more powerful options.
2 comments