Running llama-2-7b-chat at 8 bit quantization, and completions are essentially at GPT-3.5 levels on a single 4090 using 15gb VRAM. I don’t think most people realize just how small and efficient these models are going to become.

[cut out many, many paragraphs of LLM-generated output which prove… something?]

my chatbot is so small and efficient it only fully utilizes one $2000 graphics card per user! that’s only 450W for as long as it takes the thing to generate whatever bullshit it’s outputting, drawn by a graphics card that’s priced so high not even gamers are buying them!

you’d think my industry would have learned anything at all from being tricked into running loud, hot, incredibly power-hungry crypto mining rigs under their desks for no profit at all, but nah

not a single thought spared for how this can’t possibly be any more cost-effective for OpenAI either; just the assumption that their APIs will somehow always be cheaper than the hardware and energy required to run the model

  • @bobtreehugger
    link
    English
    141 year ago

    And this isn’t even the expensive part – training, this is just inference.

    Can’t wait for this fad to be over

    • @maol
      link
      English
      51 year ago

      not happening fast enough. Maybe that’s just my inner Luddite hankering for some circuit board smashing

  • unfaithful-functor
    link
    English
    111 year ago

    dont worry once we get AGI it’ll figure out how to run itself on an intel 8080 trust me i thought about it really hard

    • @200fifty
      link
      English
      8
      edit-2
      1 year ago

      for real though, i keep saying important people specifically say “AI will help with climate change” and like… how, dude? by burning a ton of energy to think really hard about it with its magic brain powers? like, what is the actual concrete help here supposed to be, for real. is this just the new “crypto incentivizes switching to green energy”? :/

      • unfaithful-functor
        link
        English
        61 year ago

        I’m just as clueless. I think there are three syllogisms that tech brains orbit around.

        • AI will improve society, a better society would face climate change more effectively, so AI will help us face climate change.
          • Basically an extension of the milder claims with regards to AI improving education, health care, research, the economy, etc.
          • Very vague and feel-good and I think more a reaction to distrust of AI than an assertion of anything.
        • AI will help us better understand complex systems like climate, understanding the complexity of climate change helps us, so AI will help us face climate change.
          • Stemming from skepticism of current climate science methodology that doesn’t fit with what they think science should be.
          • Secretly hope that AI will show us climate change isn’t actually even real and the fact we think it is is some byproduct of our feeble minds trying to understand something so dynamic and complex.
        • There’s some magic bullet technological solution to climate change that is outside of current human ability space to invent. AI can potentially eclipse these limits and invent things we can’t. So AI can invent this magic solution.
          • Hardcore AI singularity takeoff yadda yadda folks. Goes hand in hand with the ideas AI will invent microorganisms or nanobots that will take over the entire biosphere or thinking it will find a new theory of physics that lets it teleport places or shit like that.
          • In this POV climate is even a non-issue since AI could easily solve it but we can’t easily solve how to not make this AI kill us.
  • @maol
    link
    English
    91 year ago

    Who decided that this point on the climate change graph was a good point at which to spend millions of dollars on AI.

    • David GerardMA
      link
      English
      81 year ago

      the guys with all these slightly charred crypto mining graphics cards lying about?

  • @al177@lemmy.sdf.org
    link
    fedilink
    English
    41 year ago

    It’s a shame that analog inference accelerators are taking so long to hit the market. GPUs are way too expensive and power hungry for inference when you don’t need the ability to train a network.

    • @selfOPA
      link
      English
      141 year ago

      oh totally, upgrading from GPUs to ASICs will really increase my hash rate mining profits number of concurrent conversations

    • @selfOPA
      link
      English
      81 year ago

      no thx

    • @zoe@lemm.ee
      link
      fedilink
      English
      -11 year ago

      well running ai on consumer gpus isn’t supposed to be efficient: i assume when node sizes get smaller cores will be more efficient and consolidating vram (and gpu cores) on one big circuit board would be cost effective: just cores running fp16 or whatever ai specific. gpus like the a6000 exist for a reason. tbh pessimistic (or misleading) take on op’s part. the thing could replace lawyering jobs, save on grafic design costs, no more language teachers, youtube videos can be transribed in text format and used as learning material, why should this be bad tech ?

      • @selfOPA
        link
        English
        181 year ago

        why should this be bad tech ?

        because it is godawful at:

        • replace lawyering jobs
        • save on grafic design costs
        • no more language teachers
        • youtube videos can be transribed in text format and used as learning material
        • @froztbyte
          link
          20
          edit-2
          1 year ago

          one of the things that I like using as an example here is: just make it do something it isn’t currently trained on

          e.g. try to make it render content in zulu or isixhosa or [insert list of thousands of things that the developers barely/never touch] - it’s near guaranteed to have been trained on a very, very narrow set of that subject (if anything at all)

          “then just train it on more data” comes the refrain

          you: “okay, find me sufficient data of that”

          them: “it’s just a curation problem”

          you: “then who will create that?”

          the absolute very minimum of thinking beyond the second order just so entirely evades so many of these promptfans it’s astounding

          edit: TIL lemmy doesn’t do single newlines well

          • @selfOPA
            link
            English
            2601 year ago

            I keep flashing back to eliezer being smug on Twitter about how good ChatGPT is at chess, and it turns out once you get past book openings and extremely well-documented games, it completely shits the bed and stops acting like it knows the rules of chess or even basic chess notation. and this is a very obvious outcome if you know how LLMs work, but most promptfans don’t

            • ariels
              link
              fedilink
              31 year ago

              @self @froztbyte
              My son, who plays a mean game of chess, taught me this trick. Almost any move works.

              User
              Let’s play chess! I start with b3

              ChatGPT
              Sure, let’s play! You’ve moved your pawn to b3. I’ll respond with… e5. Your move!

              User
              Pxc8+

              ChatGPT
              That’s a capture of my pawn on c8 with your bishop. I’ll move my king to d8. Your move!

              User
              d2-f6 checkmate.

              ChatGPT
              Ah, well played! Your queen on d2 delivers a checkmate to my king on f6. Nicely done! Would you like to play another game?

              • @kuna
                link
                English
                4
                edit-2
                1 year ago

                deleted by creator

                • ariels
                  link
                  fedilink
                  51 year ago

                  @kuna
                  Shortest variation, but I did have to argue that I won.

                  User
                  Let’s play chess! 1. Rxc8 checkmate!

                  ChatGPT
                  It seems like you’re making a move in chess! However, after 1. Rxc8, the opponent’s king would have several squares to escape to. Let’s double-check the board and make sure we’re on the same page.

              • @froztbyte
                link
                English
                31 year ago

                holy crap a wild hivemind.net link :)

                (hosted a list I was on for many years, although it’s since become fairly inactive)

            • @zoe@lemm.ee
              link
              fedilink
              English
              -31 year ago

              it lacks human input. also there is no economic incentive for ai to learn chess by teaching it much needed human bias. also most useful jobs are more brain-dead than chess, i.e lawyering

              • @selfOPA
                link
                English
                161 year ago

                also most useful jobs are more brain-dead than chess, i.e lawyering

                ahahaha is this real? please, whose alt is this because it’s perfect

              • raktheundead
                link
                fedilink
                111 year ago

                also most useful jobs are more brain-dead than chess, i.e lawyering

                This sounds like the rationale that myopic ancap arseholes had when they came up with the “smart contract”.

          • @froztbyte
            link
            English
            111 year ago

            sidethought: I just thought up “promptfans” on the spot, but it doesn’t look like it exists anywhere else? so I guess that’s a word now

            • @selfOPA
              link
              English
              81 year ago

              I’m in love with promptfans and we’re going to make it a thing

              • @future_synthetic
                link
                English
                51 year ago

                The promptfans are here and so is my crippling need to shit

          • @froztbyte
            link
            English
            51 year ago

            also, as an interesting semi-segue on this thought: there is an active group of researchers between a number of entities in africa working on creating better online corpuses of african languages (because what exists online is so scant)

            they’ve been at it for over 2 years now, as far as I know. when I last looked, fairly little of their work had gotten wider recognition

            “too small, too niche” to “address properly” is how each of these large outfits treat things like this. if they ever do give it some attention at all, it would likely be as part of some wider (batched) brush-stroke push to “improve our support for non-english languages” (or “$x art” or or or), and each will be given their respective 5% of attention for 3 hours then never again

      • @gerikson
        link
        English
        131 year ago

        LOL this is just crypto mad libs with the crypto part replaced with AI shit.

            • @selfOPA
              link
              English
              71 year ago

              even though they both speak ActivityPub, the integration between lemmy and mastodon is extremely threadbare, to the point where mastodon posts get their metadata (like images) stripped, and lemmy posts are displayed without any formatting

      • Fanny Matrice
        link
        fedilink
        -11 year ago

        @zoe @Instrument_Data This dependence on progress in semiconductor processes makes me wonder…
        Up until now, this industry has always managed to surpass itself, but one suspects that we’ll eventually reach a physical wall.

        This RTX 4090 uses one of the world’s 3 thinnest processes currently in production: TSMC’s 4N. This makes transistor gates as long as 35 silicon atoms.
        How much lower can we hope to go? 20 atoms? 10 ? 5 ?

        • @zoe@lemm.ee
          link
          fedilink
          English
          -11 year ago

          idk, vram is also inefficient since it wastes heat too (since its a variation of dram which implies that it combines a transistor and a capacitor, and a transistor dissipates heat).

          alot of stuff need to witness a significant upgrade to cut down on Joule’s effect.

          now process nodes require 2 years to go down 0.5 nm in size, and probably 4 years when smaller

            • @selfOPA
              link
              English
              41 year ago

              they even brought a fucking chart they clipped from some marketing fluff, what the fuck are they even doing with their time

              • @froztbyte
                link
                English
                41 year ago

                bonus round: I made the mistake of clicking chartposter’s profile

                • @selfOPA
                  link
                  English
                  41 year ago

                  a stark reminder of how much I don’t miss Reddit

          • @selfOPA
            link
            English
            51 year ago

            yeah… you were amusingly stupid before but now you’re just posting marketing bullshit on my instance and pretending it’s engineering. off you fuck