• @anton@lemmy.blahaj.zone
    link
    fedilink
    English
    83 months ago

    If you change the tokenizer you have to retrain from scratch, but you can do so with the old, unpolluted data.

    It’s genius if you think about it,* you can waste energy and tell your investors it’s a new better model, while staying upstream from the river you pollute.
    * at least for consultants, compute providers and other middle men.

    • @UnseriousAcademic
      link
      English
      43 months ago

      I remember one time in a research project I switched out the tokeniser to see what impact it might have on my output. Spent about a day re-running and the difference was minimal. I imagine it’s wholly the same thing.

      *Disclaimer: I don’t actually imagine it is wholly the same thing.

      • David GerardOPMA
        link
        English
        43 months ago

        there’s a research result that the precise tokeniser makes bugger all difference, it’s almost entirely the data you put in

        because LLMs are lossy compression for text

        • @froztbyte
          link
          English
          33 months ago

          latent space go brrrr