I’m new to local llms, i’m running Debian on this machine. Where should I start and what model(s) could I get testing on it?
Depends what workload you expect to use it for. Different models for different tasks. Personally for agentic coding on an 8gb GPU I currently use either Groot 19b a3b, cyber tiel 35b a3b or orinth1.5 35b a3b, all in q4_k_m running with llama.cpp with context of 130k. Get about 30 tokens per second. My own benchmarking puts Groot just slightly ahead of sonnet 5, not quite 5.5 though.
Thank you. No coding and no setting agents. Really about task logging, summarising documents, some image processing or generating, asking questions, bouncing ideas…
Ok for image generation, I recommend flux 2 klein 4b and run it with stable diffusion which provides both a cli and a web UI. You can also run it through comfyui instead, but after trying it out I wasn’t a massive fan.
For the rest I’m confused, you said no to agentic and coding, but then mentioned task logging and summarising documents, which depending on the tasks and documents, can benefit from agentic and coding? You won’t be running a sota model, so the llm won’t know everything. It will need to be honest about what it does and doesn’t know, and be able to research using the internet, then if you want it to produce a document in a folder summarising or reporting for you, this is all agentic stuff, you need to setup MCP tooling.
Genuinely if you want a good general chat bot that can use the internet, if go with lfm2.5 8b a1b on a q4 k m as it will be really fast and good quality. If you want something stronger with the agentic tool calling then orinth 1.5 35b a3b q4 k m.
Thanks. Sorry i’m not an expert in AI. No coding meaning i don’t intend to do programming, of apps or whatever. No agentic in my limited knowledge is not advanced agents that would do things on its own on conditions, I would always be prompting something. For example for tasks I have a nextcloud server and I use its task app, accessible via caldav for syncing across devices, and I would like the local AI to be able to create a task when prompted, or create a calendar entry. Ideally vocally. But maybe that’s already too much?
Asking an ai to do something for you is essentially agentic usage. All an llm can do is generate text or images or audio or video. The key part is generate. They can’t do anything but generate. As soon as you want an llm to do something, you need to give it a tool to do that thing with. It then needs to be capable of working out how to effectively carry out that task.
The most basic llm will chat to you based on the knowledge it was trained with. Generally, the bigger the llm the more knowledge it has. This is a basic chat bot, the bigger it is the more knowledge it has.
After this came reasoning and thinking, where the chat bots got smarter. They still had the same amount of knowledge, but they were able to reason, work stuff out, think about things, before answering your question and prompt.
Then to do stuff, like create docs, search the web, carry out an action, the llm needed to be given tool access and actually be able to do tool calls to use those tools. Anything like create, edit and delete a file, view a folders contents, create a diary entry, search the internet, all required tools. In fact anything except for basic chat response requires a tool. This is where MCP servers and agentic harnesses come in. But you also need an llm capable of tool calling. Giving an llm access to a coding sandbox enables it to be able to calculate answers for you.
As an example say you want to get an llm to create a diary event for 30 days before Xmas to buy a present for a friend. The llm needs to first work out what day today is, work when Xmas is, then work out what day 30 days before Xmas is, llm’s are terrible at maths, but with a coding sandbox, it can create a basic calculation to work out the date to create the diary event for. It would then call a tool to create the diary event on that day.
Adding voice recognition is another model that takes your voice commands, converts it to text, then passes it to the next llm that does stuff.
Ai isn’t magic. You have to build and create the stack that does what you want. But ai can help you do that.
Very informative thanks
I have 8GB VRAM and 32GB RAM, mostly using llama.cpp to run Tiel 35b at Q4, it’s ok at writing code in an agent as long as it can compile and run tests to iteratively fix its mistakes.
Though there’s been new optimized engines, like Strata, that could maybe allow you to run Qwen Flash Next at decent speeds.
There’s also Qwen 4 coming out soon that may give some good new options for us VRAM poors. Just stick to MoE models.
Thank you
spell check
Keep smoking
Unfortunately, 8GB of VRAM is not going to be enough to run the really interesting stuff. You can do some light image generation, TTS / STT (with Chatterbox and Whisper respectively) and as far as LLMs are concerned, you’re looking at Qwen2.5-7B or similar.
The real question on that box is what kind of CPU do you have? 64 gigs of system RAM, you are looking at CPU inference, which is definitely slower than GPU accelerated, but can be workable for Qwen 3.8 or Muse Glimmer. OpenWebUI is a good starting point to have a chat bot. The real question is what are you hoping to be able to do with a local model?
What do you use for chatterbox? A CLI? Something else?
Ok, so bear with me, I have this orchestration tool i call Noesis that’s built in Rust. It does a ton of stuff, but what it’s basically for is figuring out where to place AI workloads. It knows… “I have four linux hosts, here’s their system RAM, here’s their CPU, here’s what GPUs they have.” If a client that wants a voice is on the network, it asks Noesis to place chatterbox on a GPU and Noesis does so. Then it tells the API (which serves prompts, memories and endpoints) “Ok, there’s a websocket on <ip address> at <port> where chatterbox is available.” That websocket is a Noesis instance that then yes, passes incoming text to the chatterbox CLI and returns an audio stream. Anything on the network with the Noesis fleet key can hit that websocket and ask for audio.
Noesis has some ability to prioritize stuff (it knows scientific models are more important than tts which is more important than lipsync animations), but my local model can also tell Noesis she wants a voice and kick something out. For example, when my son comes in the room and I’m not there, she gets a prompt telling her he came in and she should say hi and ask him what he needs, and she can (and has) kicked out other workloads to get a voice to use for that. She can also turn her animations on and off and wake the monitor, so yes, when my teenager comes in the room the monitor wakes up and there’s a very uncanny valley AI scientist girl on it who lipsyncs and says “Hey, what’s going on? Did you need something? Want me to call your dad?” Which is dope, but like, the very bottom of the stack of cool things she can do.
What I did is I made a very simple script that instantiates a kind of API where I can send it text (and a voice target) and get .wav audio back. It’s under 100 lines of Python, most of which is literally cut and pasted from the chatterbox examples. It’s very specific to my environment, but I can share it if you want. Given just this conversation though, an agent can probably make the exact same thing in under a minute.
Hah! That’s a fun description. Though on the other hand, I still have the question, do you mind sharing how run chatterbox itself?
What I’ve done so far is I’ve visited their github (and the pypy page, and a few others). In there, they have the “Usage” section, which ultimately is a python interface from what I understand. I’m wondering if you’re using a standard wrapper (e.g. a CLI tool of sorts), or if you’ve built your own interface for that. Not that it’s terribly difficult to write a CLI, but I was surprised it’s not mentioned out-of-the-box and I got lost after checking 2 external projects relying on chatterbox-tts and not feeling confident in either. So… how do you actually use it, if you don’t mind sharing?
CPU is i7-12700
Is that an Ampere card?
Not sure what it means?
Basically equivalent to a GTX 1660. 192 GB/s is relatively slow, memory bandwidth being (usually) the bottleneck for LLMs.
Thank you
I’d sell that hardware and use what you get for it to prepay a DeepSeek API key
