Running llama-2-7b-chat at 8 bit quantization, and completions are essentially at GPT-3.5 levels on a single 4090 using 15gb VRAM. I don’t think most people realize just how small and efficient these models are going to become.
[cut out many, many paragraphs of LLM-generated output which prove… something?]
my chatbot is so small and efficient it only fully utilizes one $2000 graphics card per user! that’s only 450W for as long as it takes the thing to generate whatever bullshit it’s outputting, drawn by a graphics card that’s priced so high not even gamers are buying them!
you’d think my industry would have learned anything at all from being tricked into running loud, hot, incredibly power-hungry crypto mining rigs under their desks for no profit at all, but nah
not a single thought spared for how this can’t possibly be any more cost-effective for OpenAI either; just the assumption that their APIs will somehow always be cheaper than the hardware and energy required to run the model
@zoe @Instrument_Data This dependence on progress in semiconductor processes makes me wonder…
Up until now, this industry has always managed to surpass itself, but one suspects that we’ll eventually reach a physical wall.
This RTX 4090 uses one of the world’s 3 thinnest processes currently in production: TSMC’s 4N. This makes transistor gates as long as 35 silicon atoms.
How much lower can we hope to go? 20 atoms? 10 ? 5 ?
idk, vram is also inefficient since it wastes heat too (since its a variation of dram which implies that it combines a transistor and a capacitor, and a transistor dissipates heat).
alot of stuff need to witness a significant upgrade to cut down on Joule’s effect.
now process nodes require 2 years to go down 0.5 nm in size, and probably 4 years when smaller
are you two lost?
they even brought a fucking chart they clipped from some marketing fluff, what the fuck are they even doing with their time
bonus round: I made the mistake of clicking chartposter’s profile
a stark reminder of how much I don’t miss Reddit
yeah… you were amusingly stupid before but now you’re just posting marketing bullshit on my instance and pretending it’s engineering. off you fuck