Excerpt from a message I just posted in a #diaspora team internal f...
pod.geraspora.de
Excerpt from a message I just posted in a #diaspora team internal forum category. The context here is that I recently get pinged by slowness/load spikes on the diaspora* project web infrastructure (Discourse, Wiki, the project website, ...), and looking at the traffic logs makes me impressively angry. In the last 60 days, the diaspora* web assets received 11.3 million requests. That equals to 2.19 req/s - which honestly isn't that much. I mean, it's more than your average personal blog, but nothing that my infrastructure shouldn't be able to handle. However, here's what's grinding my fucking gears. Looking at the top user agent statistics, there are the leaders: 2.78 million requests - or 24.6% of all traffic - is coming from Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot). 1.69 million reuqests - 14.9% - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonb...
  • selfAEnglish
    11·
    10 months ago

    at least OpenAI and probably others do currently use commercial residential proxying services, though reputedly only if you make it obvious you’re blocking their scrapers, presumably as an attempt on their end to limit operating costs

    • db0@lemmy.dbzer0.comEnglish
      6·
      10 months ago

      Oh never heard of that. I have blocked their scrapers via agents but I haven’t felt residential proxy pain.

          • selfAEnglish
            2·
            10 months ago

            yep, it seems so! I haven’t put the permanent fix for the nodeinfo bug into place yet but it’ll be live as soon as I’m able to give it an appropriate level of testing.

        • zogwargEnglish
          2·
          10 months ago

          Infinite-garbage-maze does seem more appealing than “proof-of-work” (the crypto parentage is yuckish enough ^^) as a countermeasure, though I would understand if some would not feel confortable with direct sabotage—say for example a UN organization.

      • froztbyteEnglish
        6·
        10 months ago

        the term of art is “residential proxy” and there’s a ton of them

        for example: it’s the flipside of Bright’s free VPN service - through Bright Data they sell people access proxied via some user’s connection