everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions

  • Typewar@infosec.pub
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    9 hours ago

    I have a dumb question… what is preventing the crawlers from just eating the shit and just burn though the energy to get through the computational task?

    • SavvyWolf@pawb.social
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 hour ago

      It’ll still slow them down and reduce load on your server. I also think many of these crawlers focus on volume; time spent computing the hash is time not spent crawling someone else’s site.