I am looking at resintalling my OS (currently Mint) which I have been running for a few years.
Mint is great and I may end up going back; but was thinking of trying something new.
I want to be able to play with local models and agents; I have a pretty decent system.
- AMD 3700X
- 64 GB DDR4
- AMD RX9070 16GB GPU
- 2TB NVME SSD
- A few TB of HDD’s
What are some good options for easily running models and agents; I have been using ollama to run models; but I’m sure things have moved on since I begun playing with things a year ago.
Give me a few tips; I am going to have a bit of time on my hands over the next few months.
Look at rolling releases. I’ve used Mint but at the time (maybe still today) some projects required the latest ROCm drivers, and Mint’s repositories are just too slow to keep up with pace of AI. I decided to go with EndeavourOS since it’s a fork of Arch so it’s a rolling release, and you can keep up with the latest experimental AI hacks that are developed. The downside to a rolling release is that sometimes you update your computer and something stops working, like your sound drivers or Steam. But someone always finds a fix a few hours later. It doesn’t happen every time. I generally update my system when I need to, so about every month or two. I use EndeavourOS as my daily driver actually, so it’s not unstable at all. Anyway, look at rolling releases that have more than 1 maintainer. Look at their forum, see how helpful it is, etc.
I used a mix of Ubuntu, Fedora, and Windows for a decade or so but recently switched over to CachyOS and it’s pretty nice, installing / updating graphics drivers is easy as and having access to the AUR is pretty neat.
Switching to a new distro would also be a good time to explore better alternatives to ollama too, plain llama.cpp is decent (got it running on my laptop rtx 3060) and Strata is quite neat for supported MoE models - runs a bit faster than llama.cpp on my oculink 9060 xt + 64gb ddr4.
How is switching OS related to local models? Are there any OS that is more efficient? Is there some software that serves certain OS better? I’m curious because I’m also using Mint, have been for about a year now after ditching Windows and I was actually exploring potential benefits of other distros for the first time earlier this week. I’d love to learn if there is something to keep in mind, or is it related to your AMD hardware?
I was going to do a reinstall; it has been a few years of inplace upgrades and a lot of extra cruft that inevitably gets built up. I was asking mainly to see if there was a clear recommendation for which system is best for experimenting with local LLM’s etc…Looks like there isn’t any specific advantages to any system so I’ll probably just stick with Mint, likely wait until Mint 27 is released before doing an upgrade.
Ah, I was worried that there was some local model distro meta that I wasn’t aware of. I have no knowledge of AMD but according to DeepSeek we seem to have somewhat comparable hardware for local LLM. I just noticed the 64GB RAM and figured you’d be on a whole different level, but apparently that setup is in the mid-high tier, spanning from 7-32B? What kind of models have you tried so far, and did any stand out? I’ve only been able to get a 13B Qwen3.5 model as most myself (which was way too slow to be worth it). From what I have picked up >24B is where you get a generally good model that goes beyond a niche purpose.
It has been a while, but last model I ran was gemma4:e4b 9.6GB. It ran very quickly; also have run deepseek-coder-v2:16b 8.9GB, again very fast…
I am looking to get back into it once I have a bit of time on my hands; I’ll look into the state of the art models again and see what I can run.
I would say that the best os/distro is the one you’re comfortable with. With that said, I would really iron out why you want to switch, like what you’re looking for. Stability? Debian. Newest features asap? Arch. Mix? Fedora. General support? Probably any of them, but fedora or Debian if you want properly supported packages and not rely on the AUR. (You can use any of the derivatives, but I can’t say they do anything special for me, so I stick to the core distros)
