• yeehaw@lemmy.ca
    link
    fedilink
    arrow-up
    5
    ·
    5 months ago

    How does that work, does it go out and start indexing the internet for you?

      • lud@lemm.ee
        link
        fedilink
        arrow-up
        3
        ·
        5 months ago

        That’s a shame, It would be more interesting to run your own crawlers.

        (Yes I realise that would be computationally intense and hard, but it would nonetheless be cool)

        • whaleross@lemmy.world
          link
          fedilink
          arrow-up
          3
          ·
          5 months ago

          There is probably a neat wget oneliner that could crawl everything on the open web. The real challenge is how to index all the information. That might be a neat Perl oneliner.