• yeehaw@lemmy.ca
      link
      fedilink
      arrow-up
      5
      ·
      5 months ago

      How does that work, does it go out and start indexing the internet for you?

        • lud@lemm.ee
          link
          fedilink
          arrow-up
          3
          ·
          5 months ago

          That’s a shame, It would be more interesting to run your own crawlers.

          (Yes I realise that would be computationally intense and hard, but it would nonetheless be cool)

          • whaleross@lemmy.world
            link
            fedilink
            arrow-up
            3
            ·
            5 months ago

            There is probably a neat wget oneliner that could crawl everything on the open web. The real challenge is how to index all the information. That might be a neat Perl oneliner.