You might remember a company called Google. In 1998, they came up with an amazing product — a web search that didn’t suck! And it could find things! With the state of search in 1998, this was incre…
  • smeenz@lemmy.nzEnglish
    2·
    3 months ago

    Then the user agent string will just quietly become randomised so you can’t match it reliably because it turns out that honouring robots.txt was always little more than a “gentleman’s handshake”.

    • David GerardOPMAEnglish
      3·
      3 months ago

      this is a problem we have had for a while now, i assure you

      • EvinceoEnglish
        3·
        3 months ago

        Yeah an adversary like Google isn’t something you can easily block without really annoying legitimate users unfortunately. Nothing is stopping them from turning every chrome instance into a botnet node except for the angry article that would run in Ars Technica.