You've probably heard me complain about the "AI crawlers" before, but now I actually have some hard numbers I can put up to show their im...
  • dumnezero@piefed.socialEnglish
    12·
    2 months ago

    Enter… your TV?

    Yes, I posted about this weeks ago. The fucking “IoT” can be a fucking botnet. I fucking hate it.

    Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.

    sigh

    In terms of what we’re doing, we’re turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it’s a necessity.

    Yep, this is a good idea.

  • moonleay@feddit.orgEnglish
    6·
    2 months ago

    Okay, I’m gonna make myself very unpopular here, but I feel like they got what was coming for them. I think nobody should feel bad for them.

    Popular figure heads of the kernel and the linux foundation itself embraces and uses generative AI. (For code & graphics).

    They lost their right to complain about scrapers, when then started using the result of them in my opinion.

    (It is also quite ironic that the author of this blogpost has an AI pfp on his fedi profile.)

    • andioop@programming.devEnglish
      1·
      2 months ago

      The highminded part of me says it’s bad that this is happening, especially given the freedom Linux still provides from the other, more locked-down options. And that you may still choose to be a maintainer or contributor without being all-in on AI, and it sucks to impact “innocents” that way. And at least what I assume is Torvalds being judicious and not purely vibe-coding is better than people who type “make me a kernel” and expect the world.

      The petty part of me says “okay, your project won’t be part of the anti-AI ideology, glad to see your project suffer because of it then, even if just in this way.”

      • moonleay@feddit.orgEnglish
        2·
        2 months ago

        Yes, its a bit petty, but I can’t feel like this is not a “oh, no, consequences of my actions” moment.

        Use a machine, which was trained on the stolen knowledge and art of the whole of humanity? Yes, please.

        Then feel the pain when the corpos scrape your data? No. None of that.

        Nobody would scrape websites that hard, if people would not use generative AI.

        • starelfsc2@sh.itjust.works
          1·
          2 months ago

          I mean didn’t they say AI code is allowed, not that you should use ai? Those aren’t the same thing but I mean sadly I kind of agree, if you pay (or allow others to pay) corporations you know are evil, and then they’re evil towards you… 🤷

  • Kissaki@programming.devEnglish
    6·
    2 months ago

    TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones.

    I’ve read similar and even worse before. Probably from Codeberg, or maybe a smaller host of Forgejo or GitLab.

    The blame html pages were repeatedly being requested. Which is even worse and even less plausibly useful than a commit html page.