Excerpt from a message I just posted in a #diaspora team internal forum category. The context here is that I recently get pinged by slowness/load spikes on the diaspora* project web infrastructure (Discourse, Wiki, the project website, ...), and looking at the traffic logs makes me impressively angry.
In the last 60 days, the diaspora* web assets received 11.3 million requests. That equals to 2.19 req/s - which honestly isn't that much. I mean, it's more than your average personal blog, but nothing that my infrastructure shouldn't be able to handle.
However, here's what's grinding my fucking gears. Looking at the top user agent statistics, there are the leaders:
2.78 million requests - or 24.6% of all traffic - is coming from Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot).
1.69 million reuqests - 14.9% - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
and the list goes on like this. Summing up the top UA groups, it looks like my server is doing 70% of all its work for these fucking LLM training bots that don't to anything except for crawling the fucking internet over and over again.
Oh, and of course, they don't just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don't give a single flying fuck about robots.txt, because why should they. And the best thing of all: they crawl the stupidest pages possible. Recently, both ChatGPT and Amazon were - at the same time - crawling the entire edit history of the wiki. And I mean that - they indexed every single diff on every page for every change ever made. Frequently with spikes of more than 10req/s. Of course, this made MediaWiki and my database server very unhappy, causing load spikes, and effective downtime/slowness for the human users.
If you try to rate-limit them, they'll just switch to other IPs all the time. If you try to block them by User Agent string, they'll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.
Just for context, here's how sane bots behave - or, in this case, classic search engine bots:
By Wyatt Reed - Dec 16, 2024Footage showing CNN’s regime change-crazed correspondent supposedly freeing a forgotten prisoner from a Syrian jail has been exposed a scandalous fraud, yet the network continues to re-air it, while defending its corresp…
Ehemaliger Richter äußert sich im Gespräch zur potenziell demokratiegefährdenden Online-Zensur auf EU-Ebene und zu den Auswüchsen der deutschen Pandemiepolitik
Das Gesetz über digitale Dienste (englisch Digital Services Act, DSA) ist seit dem 17. Februar 2024 in Deutschland anwendbar. Laut der Bundesregierung zielt es auf "ein sichereres und verantwortungsvolleres Online-Umfeld" ab. Der Richter außer Dienst und Rechtsanwalt Dr. Manfred Kölsch, der im Mai 2021 aufgrund der haarsträubenden Pandemiepolitik sein Bundesverdienstkreuz zurückgab, hat sich in mehreren Publikationen (2) mit diesem Gesetz beschäftigt und sieht vor allem in der Unbestimmtheit der angewandten Begriffe eine Gefahr für den Debattenraum. Die Möglichkeit, ganz frei zu Diskutieren, sei für ein demokratisches Zusammenleben elementar wichtig, der Streit zwischen kontroversen Meinungen gehöre untrennbar/unabdingbar/unverzichtbar zur Demokratie . Kölsch’ Meinung nach birgt der Digital Services Act das Potenzial, dass eben nicht nur tatsächlich rechtswidrige Inhalte gelöscht werden, sondern auch Meinungsäußerungen, die eigentlich von der Meinungsfreiheit gedeckt sind.
Im zweiten Teil unseres Gesprächs geht es um die illegale massenhafte Beschaffung von Masken unter dem ehemaligen Gesundheitsminister Jens Spahn, die Dr. Kölsch zufolge eindeutig illegal verlief. Der Bundeshaushalt sei hierdurch mit stolzen 5,9 Milliarden Euro belastet worden, ohne dass diese Ausgaben einen sinnvollen Beitrag zur „Pandemie“-Bekämpfung geleistet hätten. Manfred Kölsch spricht diesbezüglich von Steuergeldvernichtung und hat auch darüber eine ausführliche, sehr lesenswerte Analyse veröffentlicht.
Former Romanian Prime Minister Admits Romania Is Run By The U.S. State Department And Soros In an interview on the Marius Tucă Show, Former Romanian Prime Minister Adrian Năstase admitted that “Romania’s Internal Organization” has been run for the past thirty years by “segments of the State Department, together with the Soros Team.”
Năstase, who served as Romanian Prime Minister from 2000 to 2004, said that the “internal affairs” of his country are handled by the U.S. “State Department” and the “Soros Team.” Romania’s defense is in the hands of the Pentagon and the White House. “Right now, we fundamentally depend on them from a security point of view, especially in the context of the war next to us. There are many centers of power in the United States. In the security area, we obviously depend partly on the White House, the Pentagon, and partly on the State Department.”
Heute hatte ich ein verängstigtes Paket bei der Arbeit. 😂 Wir haben dort lange Eisen- und Plastikstangen, um Pakete herunterzuziehen, die an Rutschen feststecken und heute, als eines feststeckte, nahm ich die Stange und wollte es herunterziehen, da fing es an selbst runterzurutschen. Vielleicht dachte es: "Ach du meine Güte! Eine Stange!" 😂😂😂
PALM BEACH, FL — Inside sources said President-elect Donald Trump drew a hard line in the sand during a dinner meeting last week with Canadian Prime Minister Justin Trudeau, as witnesses reported overhearing Trump telling Trudeau that the U.S.
A Fiorentina midfielder collapsed in the middle of the game against Inter yesterday. Medical experts, as you might expect, are totally mystified and baffled as to the cause:
Fiorentina midfielder Edoardo Bove is in a medically induced coma after collapsing on the field during his team’s Serie A match at home to Inter Milan on Sunday, with the game abandoned shortly afterward.
Bove’s teammates immediately called for medical help and both sets of players surrounded the 22-year-old while he was being treated before he was swiftly stretchered off to an ambulance near the pitch and taken to Careggi Hospital in Florence.
“Fiorentina and the Careggi University Hospital announce that the footballer Edoardo Bove … is currently under pharmacological sedation and hospitalized in intensive care,” Fiorentina said in a statement.
We don’t KNOW it’s the #vaxx… Speaking of which, I’ll be sharing my theory on the true purpose behind the vaxx on a UATV-exclusive Darkstream tonight, which I suspect some of you will find interesting.
Fiorentina midfielder Edoardo Bove is in a medically induced coma after collapsing on the field during his team’s Serie A match at home to Inter Milan on Sunday.
A Fiorentina midfielder collapsed in the middle of the game against Inter yesterday. Medical experts, as you might expect, are totally mystified and baffled as to the cause:
Fiorentina midfielder Edoardo Bove is in a medically induced coma after collapsing on the field during his team’s Serie A match at home to Inter Milan on Sunday, with the game abandoned shortly afterward.
Bove’s teammates immediately called for medical help and both sets of players surrounded the 22-year-old while he was being treated before he was swiftly stretchered off to an ambulance near the pitch and taken to Careggi Hospital in Florence.
“Fiorentina and the Careggi University Hospital announce that the footballer Edoardo Bove … is currently under pharmacological sedation and hospitalized in intensive care,” Fiorentina said in a statement.
We don’t KNOW it’s the #vaxx… Speaking of which, I’ll be sharing my theory on the true purpose behind the vaxx on a UATV-exclusive Darkstream tonight, which I suspect some of you will find interesting.
Fiorentina midfielder Edoardo Bove is in a medically induced coma after collapsing on the field during his team’s Serie A match at home to Inter Milan on Sunday.
My wife has bought herself a #Samsung #Android S7. When she tries to directly open the all-known-all-famous download URL f-droid.org/F-Droid.apk she got redirected to Amazon. I had to send the file directly to her via an other app.
morph
in reply to Roland Häder🇩🇪 • • •