I’d link to some blog posts about this an example, but the site they’re from went down a while ago.
At risk of giving anyone ideas, why do LLM training scrapers request pages from sites millions of times per day instead of just doing the equivalent of wget -r https://example.com/ ? If the point is just stealing things people have written, what do they accomplish by wasting a web host’s resources beyond being an ⊛ to webmasters?
Edit: Let me elaborate. A lot of the answers i’m seeing are just restating the problem without explaining why LLM scrapers are apparently all either coded by idiots or assholes. I know that they’re hitting sites with unreasonable numbers of requests, wasting bandwidth, and making tools like Anubis too important. I know that’s what’s happening. I’m asking why. What they gain from not being even a little intelligent about this.
With all the money and effort (and maybe even brainpower) going into this, surely there’s some explanation beyond incompetence.
You must log in or register to comment.
My hypothesis is that those scrappers run on botnets hosted on infected servers, so they want to get the info as fast as possible, since you don’t know how long it will last before they are detected and removed.
Another option is that so many servers are scrapping the same web that one server may send only a request every a few seconds, but the whole fleet could be sending thousands of requests per second, some big websites can handle the load, but most small to medium size sites get overwhelmed.
I think many people tend to mistake any ddos for an AI scraper.
People are shitty. And some will just ddos you not for AI but trying to find vulnerabilities or just because they don’t like your site.
Deeper analysis of the attacks should be given to determine if they were scrapers in the sense that they were trying to get the information from your site, or they were just a ddos trying to take the site down, or maybe they were proving trying to find a vulnerability.
The latest is the one I get the most. I get petitions for over 400 paths in less than a second trying to find a vulnerable endpoint.
In these cases they do it so quick, that might provoke a ddos in some servers, because they know that they will get ban as soon as the log get processed, so they need to act quick.
You are sitting behind a receptionist’s desk with three coworkers. You need to check people into your office for visits.
Because there are four of you, and people are generally good about queuing, your normal operations run smoothly.
Now fifty people walk into the building at once. They don’t really care about checking in, or even visiting the office. They want to know if you have free coffee, a public bathroom, your elevator inspection on file, the date of your last fire inspection, and dozens more inane questions that aren’t what you’re used to handling. They refuse to wait in a line, they’re talking over one another, and if you take more than ten seconds to answer, they walk out of the building but come back to bother you a few minutes later.
Does your office run well, and can you check in a legitimate visitor in a timely manner still?
I would have said the 50+ new people would queue up in a line, but without giving the receptionist a toilet break. The receptionist would be so busy with them that the he can’t afford to leave the counter and ends up peeing on the spot, causing the building to be shut down due to hazardous materials.
There’s a lot of things. Some scrapers aren’t smart and will grab every link they see, just in case. Those links will often be redundant, but the scraper might not know that until it’s done downloading it. A 1MB download doesn’t seem too hard on the server, but do that a thousand times a minute and that’s a decently heavy load on any small website. I saw a creator talking about how they noticed AI bots scraping their website since they had literal decades of code that was well documented and guaranteed AI free. The bots could easily grab the git files and have all of that code with every commit message for about 4GB. But instead the AI was asking the Website to render each and every commit as a webpage which took a 0.5KB change and rendered it as a 5MB webpage since it showed all the code before and after, all the formatting for a webpage, etc. This meant that each scraper was downloading TERABYTES of data from his website when all it wanted was in a nice, single download that was in a very computer-friendly format to parse. None of these scrapers were made smart. If they were, we wouldn’t notice them. It’s because they’re dumb, but spending a lot of time and money on “avoiding detection” that it’s dangerous.
Legit I had a website give me a 403 on an API request earlier. They had changed their API to block anything trying to masquerade as a browser. They actually just wanted legit API users to send a user agent that had a username and a project name. If you pretended to be a browser, you got blocked. On the 403 page, they had a direct link to download their whole database in one go if that’s what you wanted. They had links to the database on megaupload that was updated every 24hrs, and then a patch database that listed all the recent changes to keep you up to date. Literally they don’t care if the data is taken. What they cared about is wasting their bandwidth and trying to be sneaky about it.
MusicBrainz?
A lot of inaccuracies in your post. These crawlers use real browsers so they cache all “redundant content” just like your browser does.
Non-expert here, so conjecture warning.
Loading the info from the page once is fine for training one LLM at one point in time. But there are a bunch of different companies and people doing that, and they probably keep doing it again because they want newer information.
And training is only one aspect of AI scrapers. You also have agents constantly doing websearches and summarizing, synthesizing the info at user request, all day every day.
Let’s not forget that the LLM scrapers refuse to follow the protocols that are already set out for scrapers to be able to reasonably get the content they want.
The main issue with AI scrapers is that there isn’t just one, there are dozens or hundreds of them, all doing the same thing. Most websites would handle being hammered by one or two bots just fine, but crash under the load of dozens of bots constantly sending requests.
On top of that, many of them are very badly (vibe)coded and get stuck in endless loops, requesting the same pages over and over again. At my job we IP-blocked an aggressive data-harvesting bot from our service, and the thing got stuck requesting the same page over and over again every 8 seconds, despite receiving the same error code every time. This continued for weeks.
A whole eight seconds? Talk about lucky. We have misbehaving bots that attempt to hit the same page or small handful of pages on our site 50-100 times simultaneously, repeating this every couple of seconds. And they have tons of IP addresses they do it from so you don’t see the same IP in your logs more than a couple of times a day. They seem to be doing it specifically to attempt to evade simple IP detection in order to make it harder to block them. And there aren’t that many IPv4 addresses in the world anymore, so the conspiracy theorist in me wonders if the AI chucklefucks are using some manner of malware to route their traffic through botnets these days. Stealing people’s bandwidth and not just their copyrighted content. It’d certainly be on brand.
For my block lists I have to look at patterns of behavior now instead, and even that’s not a sure fire thing. It’d be easy to block a legitimate user who just so happens to be trying to load the same page/product at the same time the clankers are fixated on it.
Putting the entire thing behind Cloudflare helped a lot. Unfortunately, I just got an email from them which seems to imply that by default they’re going to start letting more bots through, which seems like a counterproductive move…
I do wonder about Cloudflare, and the conflict of interest of them being an AI company and an AI defense company at the same time.
the conspiracy theorist in me wonders if the AI chucklefucks are using some manner of malware to route their traffic through botnets these days
Not a conspiracy at all: it’s a whole industry called “resnet proxies” that considerably predates LLMs.
There’s a whole good, bad, and ugly to it; they’re invaluable as a privacy and public-access tool, but also great for blackhats and ticket scalpers. Similarly, the proxy nodes come from a full spectrum of sources ranging from normies getting paid kickbacks to plug a box into their router, to pseudo-ISPs serving ghost clients, to malware farms, and everything in between.
Everybody in this thread is talking about what they’re doing, but not a single reply has been able to explain why the bad behavior somehow benefits the companies doing it.
I don’t believe the only reason is incompetence; there’s got to be somehing else to it.
The unfortunate answer is because it’s cheaper to not give a shit. Sending a request and waiting for a timeout costs next to nothing, and scales linearly in terms of compute cost. The overwhelmed server on the other end slows exponentially with each concurrent request. The crawlers are set to maximize the efficiency of local resources, which include both wall-clock time and developer time. Why send one request at a time when your server can handle tens of thousands?
Try x; wait 60 seconds, if fail: put on a list to try again later.
Costs nothing to write and nothing to run. And if you own the hardware, and are paying for power already, may as well extract maximum dollar per watt.
To lock all knowledge in their models as the onky source, destroy independent websites , destroy physical books after scanning , destroy peoples brains
They’re not destroying the books “after” scanning ‘for the evulz’ to deprive the world of them. They’re destroying them before scanning because cutting the spines off with a bandsaw lets them drop the stack of pages in a sheet-feed scanner and get the job done a lot faster than using a book scanner.
Similarly, I’m sure they’ve got some actual rationale for their crawler behavior that makes sense (at least to them) and isn’t a conspiracy theory.
Nobody is a mustache-twirling villain in their own mind, even if they turn out to be so from everybody else’s perspective.
No they’re just stupid when it comes to the crawlers. They have VC money and no expectations to behave well. They want to capture everything they can as fast as possible because the race to build up datasets yup use in training is the only real target they have
It’s 100% a mix of various incompetences. I develop anti fraud systems and deal with cralwers all the time. The web is actually really complex, probably the most complex technology in history of human kind and I’m not even kidding.
I don’t know, man. Almost everything in the last 10 years can be explained simply by incompetence.
No, quite a lot in the last 10 years has very much been caused by hate, malice, and greed.
It’s not incompetence. They just don’t care, the same way they stole all the books from the internet.
A web server can only handle so many requests a minute, and the bots are making somany legitimate users can’t get through
i may be wrong, but it seems like most scrapers are still sane, but there’s been a huge increase in the total number of scrapers and some are insanely intense. maybe they’re “vibecoded”…
The latest scrape I saw that took down one of my sites just threw shit at the wall until something stuck. It took one second and did 50+ requests mostly for pages that didn’t exist.
Because a -r is likely what they are doing. Recursively checking everything every 3 microseconds
So they’re just checking as fast as possible to get every update as fast as possible?
This still sounds like idiot design, scraping hard enough to take down sites. It’s like cutting open the goose that lays lead eggs in the hope that you can get more lead and convince people that lead is better than gold.
While this is idiotic and counter productive long term, the thing to remember, it’s not just one scraper, it’s hundreds of them, simultaneously trying to outscrape the other, each trying to hoover up as much as possible as fast as possible.
If that’s not enough, the Facebook scrapers hit the same URL over and over again, even when that URL returns an error. Their approach to web scraping is essentially: “Fuck you, deal with it”, even if the website is run by a small business that cannot financially justify more bandwidth or server capacity.
I’ve seen this process bring sites down, to the point of actively having to block all non-human requests, just to be able to serve customers. This of course doesn’t work long term because the bots continue to probe and morph.
Welcome to reason 2 to hate the current bubble.
As soon as a page loads, the AI goes to a new page. Not just one new page, every link on the page.
It’s functionally a ddos attack. Because it quickly spirals exponentially. The limit isn’t how fast the AI can scrape, it’s how much bandwidth the website server has.
But isn’t this how ordinary search crawlers and tools like wget also work? I’ve never heard of them causing these same problems.
Ordinary search crawlers are throttled and they do stuff like respect your robots.txt and sitemap.xml (directives that tell the crawler what links to scan or not and how frequently). These naive crawlers hit everything at max speed, and on a content-heavy site like lemmy that can easily take out a server.
Why do the AI companies’ crawlers work differently from traditional ones?
It’s not necessarily the big companies themselves doing it? But, there’s a push right now to scrape as much data as possible to feed the training of the AI models, and if your business model is entirely based on selling that data you don’t really care who you’re going to piss off. So, in turn these new scraper bots basically behave in the same way that attack bots do where they’ll be behind VPNs and switch IPs and stuff if you block them.
This is incorrect as you don’t need vertical scaling for broad crawling which by definition is a horizontal operation. So you can slurp the web with a per domain throttle that has a respectful limit. The only pressure is browser cache time as if you’re crawling a domain you want to take advantage of as much cache as possible but modern websites have very long cache times, like days.
So it’s mostly incompetence. Crawling is actually really hard because web is just very complex.
The entire point here is that the bots are being disrespectful.
I’m just correcting you architectural definition. No crawler needs to vertically scale as you define it here and in fact it would be a worse tech for “slurp all data” goal.
The real answer is much more simple - incompetence. The inability to write a good cralwer by leaders in software tech is much damning than anything else. It’s like leading car manufacturers not being able to make the wheels round - it’s just sad.
However this actually shows a much more important problem with the web - we have no way to defer or distribute negative externalities and people have been trying to solve this since the inception of the internet itself. It’ll never be solved but minimized to the point where the impact is just something you tolerate because it’s simply impossible to solve and anyone who tells you otherwise is selling you something.
Source: I’ve been working on similar problems for the last 20 years at least.
But WTF is the point of scraping the same data over and over multiple times a second before it even has a chance to change? That’s just a waste of resources even on the scrapers’ part, because that bandwidth could be used grabbing some other new page instead!
They’re not trying to do that in particular, but on a site like lemmy the same content might be linked a dozen different ways:
https://lemmy.world/comment/25863752 https://lemmy.world/post/51995780/25862545 https://lemmy.world/post/51995780/25860801 https://lemmy.world/post/51995780/25858204 https://lemmy.world/post/51995780/25857153 https://lemmy.world/post/51995780/25857033
There’s canonical url tag or 3xx redirect so a decent crawler can resolve this and abort duplicate operations.
Same difference as being hit by a golf cart rolling down a sleight slope and 100 Semi’s going 100mph
The problem isn’t what they’re doing, it’s the speed and depth. There’s no concern for efficiency because they’re not paying for hardware and utilities.
Everything AI is focused on doing asuch as possible as fast as possible, with the hope optimization will happen organically to the point it becomes profitable.
But it won’t.
Correct. Google indexing your web page is rate limited for this reason, and you can even include a directive in your robots.txt to specify your own rate limit if you’d like the intervals to be longer (or shorter). The AI scrapers completely ignore your robots.txt. Except, I am certain, for abusing it as if it were a site map. Anything you list there is simply a target you’ve revealed to them. (“Hey, robots.txt says we shouldn’t crawl /foo/bar.html. That means there’s a page there! Let’s hammer it with 900 page load requests per second!”)
This is not exactly true because web browsers have caching.
Short answer: they aren’t just grabbing one page. They are hitting every page and link that is available in rapid succession.
Honestly a lot of web is just extremely poorly optimized which is not a problem when you have have 1,000 page requests per day but suddenly you get 10,000 and things start breaking.
Optimizing pages is very easy these days and a good website can serve hundreds of thousands of requests per day same as first 1,000 because caching is extremely powerful.
However, same goes for crawlers which can be really poorly written and extremely unoptimized. This is especially the case with broad crawlers like AI crawlers that crawl all pages as they use a real web browser causing more expense. They use real browsers because websites these days use client side rendering to defer costs to the client AND anti-bot system that require a browser to bypass.
Tl;dr: lots of very bad code
Edit: leave it to luddites to down vote almost 30 years of web dev experience lol
because you are clearly minimizing the impact of AI crawlers. sites don’t go from 1000 page requests /day to 10,000. I’ve seen sites going from 50~100 request per minute to 10.000 request per minute and they are mostly bots. no matter how efficient your site is, if the server cannot manage the connections it does not matter whether the content is cached or not.
And caching does not solve everything because many cloud providers charge for it, so even if you could potentially deliver them, it suddenly becomes too expensive to do so.
Nosensical. I wrote an entire paragraph on ai crawlers being bad.
