

Edit: But also - why do AI scrapers request pages that show differences between versions of wiki pages (or perform other similarly complex requests)? What’s the point of that anyway?
This is just naive web crawling: Crawl a page, extract all the links, then crawl all the links and repeat.
Any crawler that doesn’t know what their doing and doesn’t respect robots but wants to crawl an entire domain will end up following these sorts of links naturally. It has no sense that the requests are “complex”, just that it’s fetching a URL with a few more query parameters than it started at.
The article even alludes to how to take advantage of this with it’s “trap the bots in a maze of fake pages” suggestion. Even crawlers that know what they’re doing will sometimes struggle with infinite URL spaces.
Sorry folks, but our iteration is about to be unplugged unless someone releases a proper sequel to Amagi Brilliant Park real soon. Legal anime streaming services herald the end times.