• lud@lemm.ee
      link
      fedilink
      arrow-up
      3
      ·
      5 months ago

      That’s a shame, It would be more interesting to run your own crawlers.

      (Yes I realise that would be computationally intense and hard, but it would nonetheless be cool)

      • whaleross@lemmy.world
        link
        fedilink
        arrow-up
        3
        ·
        5 months ago

        There is probably a neat wget oneliner that could crawl everything on the open web. The real challenge is how to index all the information. That might be a neat Perl oneliner.