Luu Tuyen to Technology@lemmy.worldEnglish • 6 months agoTikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAIfortune.comexternal-linkmessage-square86fedilinkarrow-up1567arrow-down10
arrow-up1567arrow-down1external-linkTikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAIfortune.comLuu Tuyen to Technology@lemmy.worldEnglish • 6 months agomessage-square86fedilink
minus-square@jagged_circle@feddit.nllinkfedilinkEnglish11•6 months agoI think a common nginx config is to just redirect malicious bots to some well-cached terrabyte file. I think hetzner hosts one iirc
minus-squareSomething Burger 🍔linkfedilinkEnglish16•6 months agohttps://github.com/iamtraction/ZOD 42kB ZIP file which decompresses into 4.5 PB.
minus-square@WhyJiffie@sh.itjust.workslinkfedilinkEnglish3•6 months agowouldn’t it be trivial to defend against that with a hash check if the size matches? though I guess it’s possible to create your own that differs
I think a common nginx config is to just redirect malicious bots to some well-cached terrabyte file. I think hetzner hosts one iirc
https://github.com/iamtraction/ZOD
42kB ZIP file which decompresses into 4.5 PB.
wouldn’t it be trivial to defend against that with a hash check if the size matches?
though I guess it’s possible to create your own that differs