Same, getting ~3 trillion parameters in consumer hardware is rough.
If Nvidia has any foresight they’ll see the writing on the wall and start getting higher memory Spark style SMB inference machines, few people in the long run are going to pay retail API token costs,
The other problem is bandwidth. Even if you can run Kimi K3 locally, AI usage patterns tend to be very bursty. Having the scalability of cloud compute to run multiple agents and sessions at once makes it much more usable.
My current workflow is passing a human written spec to an agent to implement with strict coding guidelines, architectural decisions, etc. The agent isn’t making any decisions about the abstractions to use, it’s just creating the objects and test suites. So I don’t mind the slower bandwidth because I’m running the heavy agentic lifting over night with no need for human supervision.
But I fully appreciate that my workflow isn’t the norm. In fact my workflow it’s the exact opposite the AI grifters like Sam Altman are selling because it still involves a human with knowledge of the systems making different decisions.
I just wish I could buy enough memory to run one of these models locally. Specially Kimi K3
Same, getting ~3 trillion parameters in consumer hardware is rough.
If Nvidia has any foresight they’ll see the writing on the wall and start getting higher memory Spark style SMB inference machines, few people in the long run are going to pay retail API token costs,
Especially giving the lack of trust organizations rightfully have in big AI companies
Exactly, having the mustard toddler pull fable because of, who knows? That burned trust in the west’s AI companies.
And there is also the question of what they’re doing with the data you send into the models.
The other problem is bandwidth. Even if you can run Kimi K3 locally, AI usage patterns tend to be very bursty. Having the scalability of cloud compute to run multiple agents and sessions at once makes it much more usable.
My current workflow is passing a human written spec to an agent to implement with strict coding guidelines, architectural decisions, etc. The agent isn’t making any decisions about the abstractions to use, it’s just creating the objects and test suites. So I don’t mind the slower bandwidth because I’m running the heavy agentic lifting over night with no need for human supervision.
But I fully appreciate that my workflow isn’t the norm. In fact my workflow it’s the exact opposite the AI grifters like Sam Altman are selling because it still involves a human with knowledge of the systems making different decisions.