Do you host your own ML / AI / LLM? What do you use, and what do you use it for?
You must log in or # to comment.
Yes. My Actual Intelligence lives in my head, and runs mostly on coffee.
Just coffee?!? That’s cool.
Mine runs on:
- coffee
- spite
- tortilla chips
- & shame
If that’s not already on a shirt it should be
Mostly on coffee, not exclusively. Noticable amounts of spite & tortilla chips are also present, yes, but… no shame.
Nice!
I replace tortilla by “raclette” but that cultural.
Do you get many hallucinations?
Only when I’m deprived of coffee.
Would flowers work instead?
No. I’m not dead yet.
I’ll make sure to send you flowers, Algernon lol
@SuspiciousCarrot78@aussie.zone this comment is not (directly) for you, I just want it in context.
Before you report someone for breaking rule 1, please look a the context. Specifically, the username someone may be replying to.
LOL.
https://en.wikipedia.org/wiki/Flowers_for_Algernon
Looks like someone got big mad over a harmless, good natured and on topic joke. You love to see it.
Sorry they wasted your time.
Eh, its fine. Certainly better than the “I don’t like this so I’m going to report it” approach.
As we know AI stands for “An Indian”, so if you’re not from India, its actually impossible to self host.
Well, unless you manage to trap one in your basement, but that would violate human rights and hopefully also break the laws of your country.
You may be confusing Indians with gremlins (AGI). Which might explain ChatGPTs obsession with gremlins
critical security bug: if coffee is taken away my head hurts :(
deleted by creator
With sufficient coffee, mine shows considerable artifice.
Plastic flowers.
Yes. Openwebui/ollama for LLM, comfyui for stable diffusion. I just dick around with it as a toy.
I was put off by ComfyUI, seems awfully complex. How is your experience?
Any suggestions to start? I have Fooocus installed now
deleted by creator
Nope.
Yeah, I’m using qwen 31b a3b on an amd 9070xt requires a bit of cpu offloading, but still plenty fast. Using it wall llama.cpp. Combine that with some mcp’s such as ddg-search to make it truly useful by actually being able to search online.
I mostly use it for small tedious tasks with well defined inputs and outputs. For example when hyprland recently changed from their own configuration language to lua. At first I started going line by line translating my config to the new lua language until I realized oh wait this is exactly the type of thing that ML is useful for. Going from the well defined hyprland configuration language to their also well defined lua syntax. It banged it out in less than a minute with only a single mistake which I easily fixed. The mistake it made was that it forgot to translate the comments to lua. It did it in less than a minute and worked first try. Where as I had made several typos and gotten a few lines wrong when I was doing it by hand.
Not to say that I couldn’t do it. I would have gotten it done in about half an hour, but less than a minute is a lot faster.
I also used it to transform a bunch of unstructured data into json data, so that I could then use purpose built tools like jq to parse that. If I’m having trouble finding certain information. I’ll ask it to find me some resources to look at.
Basically small well defined tasks and parsing data is what I use it for and it seems to be pretty good at that.
What I don’t like is the way companies try to market it to people. I don’t believe people should be trying to summarize emails or messages from loved ones, writing essays or any other creative tasks for the most part. Translating is okay. I don’t expect a machine to be able to decide things for me or to be some filter between me and others.
I do, but I am becoming increasingly more disappointed as time goes on. Not just self hosted, llms in general. They sometimes help, but they mislead so many times and waste time that you don’t even notice. I think that’s the trap. When you succeed at a task, you become impressed but don’t notice how many times it failed doing a simple task. And as soon as you scratch the surface, you see how you would have done it differently and perhaps in a better way. Even just googling is bad. It does research for you, but it has no critical thinking and can’t decide what is better from the results it gets (other than google ranking) so it often leads you to think it did as good as you would, when it’s nowhere near as good. Every time I did the googling myself after it did, I did it much better. And I mean MUCH better. Ask it to find the app, it misses the most important ones, hallucinates a bunch, for ex. I found this to be the case with frontier models as well.
Self hosting has its benefits, but seeing how the ecosystem looks right now, concluding this is a huge bubble is inevitable. It reminds me of crypto so much. It looks rich and plentiful, but as soon as you dig a mm under the surface - nobody has tested it, it’s got a critical bug, it is overblown and there are issues with no response. No docs, no info, no nothing. For the biggest thing in technology in history, it is awfully hollow. I don’t mean it in a condescending way, in fact community is enthusiastic and very helpful, it’s just that it doesn’t live up to what most would expect.
A caveat I need to mention is I have not used it for coding - I have an irrational fear and resistance towards it, being a programmer. I just won’t touch it, even if it means the end of my career. I’m trying to be grown-up about it, but so far, I dont want to use it, for good and bad reasons.
Running qwen3.6 27b through llama.cpp.
It’s about as capable as sonnet 3.5.
I use it for light scripting, but real coding is done by cloud models.
I’m also using it as the brain for my Hermes agent. It sends me digests of news, subreddits, chats that I’d like to read but don’t have time for. It does a great job researching things on the web for me, too.
That’s a great model and it’s the one I use too.
Do you mean Sonnet 4.5?
I don’t have the rig to run it at real speeds but I’ve played with it over API. Seems pretty good.
No, it needs a lot more babysitting than 4.5 does. 3.5 was on the same level of mistakes, at least on the quants I have to use.
No. I still have no use for it and everything I use is automated without at a far lower footprint.
Yes, I got a Strix Halo machine before the RAM price hike and use it to run all my ML stuff on it.
Currently using llama-swap with llama.cpp/ComfyUI and opencode/Open WebUI as frontend.
I’m running Qwen3.6-27b, Voxtral Mini 4b, Piper and Qwen Image. Also, some embedding and reranking models.
I use them for:
- Tagging and classification of my documents in Paperless
- Home Assistant (voice assistant)
- Translations (both text and image)
- Transcriptions
- Some light coding and debugging
- Avatar/Backdrop generation for DnD sessions
What sort of tok/s are you getting on the strix?
About 200 t/s prompt processing and 10-20 t/s with MTP.
Greatly depends on the task, predictable things like code generates at 18-20 t/s. Creative writing more like 10-17 t/s.
Damn - I thought strix would do a bit better than that, for how much it costs.
Given the 27b is a dense model, I think the numbers are quite ok. Curious about the quant tho.
The cool thing about the strix is its large unified memory, but it lacks memory bandwith for compute intensive workloads. Something like Qwen3.5-122b MoE with only like 12b active parameters might run at twice the speed if it fits the configuration.
Curious about the quant tho.
Q8 from unsloth.
Something like Qwen3.5-122b
My go to model for knowledge. Definitely much faster at Q5 but it lacks the tool calling quality of the Qwen3.6 models. Really hoping we see a Qwen3.6-122b soon…
In case you missed the Ornith 1.0 release (Qwen and Gemma RL finetunes for agentic / coding workloads), they look interesting to bridge the gap until we see larger 3.6 models or a 3.7 release. I didn’t test them yet but according to benchmarks, the 35b MoE seems to be more or less on par with Qwen3.6 27b dense, while ofc a lot faster.
Yeah. Though I think theres a new strix out soon (Medusa? Gorgon? Something like that).
Its a bit like my P40. On paper, it has 24GB. But that 24gb is capped at 400GB/s and the ai compute is what…Pascal era?
AI = Good, fast, cheap - pick 2
Well compared to the strix, 400GB/s is not that bad, I think with fast system RAM and expert offloading you could squeeze quite something out of it when running stuff in the 100b-a10b regions.
Your bigger problem is going to be future software support.
I do, I use ollama. I mostly just tinker, but I use with with home assistant for a quasi Alexa like experience with the voice assistant, I use it for summarizing some YouTube transcripts in too lazy to read/watch, and I’ve tried to see how capable it is with coding.
Can you elaborate on what you are using exactly with home assistant ? And is English your primary language in that context ?
Trying to do something similar, English not primary and its a bit… Harder than it seems. Can’t figure out if it is because I’m not using English or something else. (3060 12GB BTW)
English is my primary, so that does make it easier. I use it for general conversion things, like asking it questions about the Titanic or making up a new story or something. It doesn’t work as well as I’d like yet, but like I said, it’s just an other thing for me to mess around with and change.
The other day I made a machine learning model that classifies images as either ‘a certain type of undesirable image’ (no, not porn) or ‘any other image’. It is 96.4% accurate and takes 14 ms to classify one image (using CPU only - with a GPU it could be 5x - 10x faster).
I plan to offer this as an API service that social media networks can use to filter posts.
Why would I?
I recently gave it a try with qwen3.5 and deepseek coder v2. I have a RTX3090 and these are the largest models that can run comfortably on it.
Conclusion, they are both fucking useless. Free tier claude runs circles.
Yeah :(
Were not there yet on consumer rigs.
If you just pulled the default version of qwen3.5 from ollama’s repo you downloaded a mediocre one that only uses ~6GB.
Check
ollama show qwen3.5and see if you get something like this in the result:Model architecture qwen35 parameters 9.7B context length 262144 embedding length 4096 quantization Q4_K_MThis is the default version I got when I first tried using ollama without any experience. It worked, but it’s a heavily quantized, lower parameter version of the model – i.e. it’s pretty dumb – compared to what you can actually run on your hardware.
I will check it later. I loaded whichever one cluade suggested lol
Ollama with gemma 4 for LLM stuff, coding brainstorming, etc.
Comfy ui with z-image or stable diffusion for images.
How is Gemma compared to others?
I don’t host it exactly, just use it when I don’t use my graphics card for gaming. I run Qwen3.6-35b on my 16gb vram RX 9700 xt with 34t/s. I use it as an IT advisor, admin and Linux teacher for my cachyOS gaming PC.
Do you find you can do coding tasks with it well?
I actually ran a series of A|B split tests (using GPT, Claude, Qwen 27B, Qwen 35B, GLM) on some code I’d written.
The Qwen models managed to find issues the others missed and offer useful suggestions.
Coding wise, they’re a little too eager to take the next step / be a helpful assistant, and context collapse is a real thing with them. I would say yes, they are capable, and probably even more so in the Qwen specific coding harness.
The thing is, small models can only hold so much in their latent space. If you give them a big job or free range task, they will find a way to monkey paw it. They need short leash and test gates.
Neat, what did your setup look like? You mentioned the qwen harness? Run it all on one machine?
Pretty simple. People keep going on about how useful these local models are for coding. So what I wanted to do was to create a standardized test for myself to see if that was true before committing to anything.
( I think the various benchmarks out there are a bit fluffy, so I wanted to try it against a real workload.)
What I did was throw a bunch of money up at OpenRouter and then used Roo to call in diff models, one at a time.
I gave each the same task - that is, here is a piece of code, here is my ticket, here is my repo. Investigate what you want and then do what my ticket says.
I already knew what was wrong with the code, but I wanted to see how obedient the models are at sticking to a scoped ticket and what they would find.
By far the best bang for buck was GPT 5.4 mini. It is exceptionally obedient at doing exactly what you tell it as long as you tell it exactly what to do.
It won’t go off piste if properly constrained.
I think for light - med workloads, $20 on ChatGPT is a crimal steal. Chat and Codex have a separate usage pool.
I’m also aware that this is open AI’s lock in phase where they provide the samples of crack for free to get you hooked. And, yes, they are crack dealers in every sense of the word.
Anyway, it’s good to know that with a little bit of elbow grease and some smarts, the smaller models, which could reasonably be self-hosted, could do a decent enough job if they are narrowly scoped.
You’re probably not going to be able to yeet an entire code base at them and go “figure out what’s wrong and fix it” while you snooze tho, but I think that’s probably a good thing from a human in the middle perspective.
I‘m not a coder, so I don’t know exactly. It is able to code, but I would say somebody with experience should guide it and have an eye on the results.
That Qwen 35B model is going to remain the people’s champ for a long time I think. Surprisingly capable, even for code. I hear it loops badly at Q4 quant?
Looping was a problem after reaching a certain context window size. The llama.cpp flags - -flash-attn on and looping penalties helped.
Probably that plus a higher quant solves it. Thing is most of us default to Q4_K_M as “precise enough”… and that seems to be kryptonite for the new Qwen’s.
That’s another thing with hosting AI that’s not often discussed. Sure, you can maybe run that 27B model…but if it’s at Q3_XS it’s going to be … “mentally challenged”.
I’ve heard the Gemma models with QAT are meant to be near full precision at Q4 size. Haven’t tried em yet.
Actually, on that topic - I’ve heard there’s a different architecture (RWKV), that’s supposed to be much more efficient for long context because it uses an entirely different KV system.
Sadly, there are few RWKV native models and retraining a standard transformer to RWKV seems like a pain in the ass. I’d need to hire a cloud GPU, distill into a different architecture, mess with datasets … honestly ICBF.
Yeah, a higher quant would be nice, I actually try not to go below Q5, but you can domino’s so much with 16GB of VRAM and the ddr4 system RAM.
But I must say I‘m pretty impressed by Qwen3.6-35b, not only from its capabilities but also from hardware requirements. MoE for the win I guess.
RWKV sounds interesting, have to look into it, thanks!
No, I have taste.
