End of 2025 - Announcements - Forums : Best Ever Albums

albummaster

Janitor

Gender: Male

Site Admin

Just a short unplanned update as we approach the end of 2025 to say a huge thanks to everyone who visits the site. Since the site's 20th anniversary, many updates have been made and and another one just behind this update (taking a bit longer than planned as lacking the finances to finish it for the time being - it's not a lot of money but broke right now).

A lot more is planned for 2026, but unfortunately for every legit user here, there are a huge number of AI crawlers on the site most of the time, which are sucking bandwidth from real users (which has been an ongoing problem for months). This is costing the site proper money in terms of the extra (unnecessary) server resources need to feed the AI machine and gives the site nothing in return. Sites like this will die out in fairly short order if AI is allowed to harvest content, present it as their own and pay for none of it (meanwhile reaping subscription fees from their users for said content) (sorry for rant).

On the plus side, overall site activity has picked up year-on-year (not including AI leeches), but need to brainstorm some ideas to combat AI as it remains an existential threat that isn't going away.

AM

Last edited by albummaster on 01/03/2026 07:12; edited 1 time in total

Romanelli

Bone Swah

Gender: Male

Location: Broomfield, Colorado

Moderator

I have, like, zero tech knowledge, and have no idea how to actually do it, but I have heard that it's possible to block AI crawlers from being able to scrape your site for content. I did do a quick google search and saw that there are multiple sites that offer solutions. If you haven't already, maybe give that a try?

I hate AI very much. Also, Sarah Connor was right.

Thumbnail. Click to enlarge.

_________________
I'm leaning on the threshold
Of her mystery
And crashing through the walls
Of dying history

MadhattanJack

Just to end the list

Gender: Male

Romanelli wrote:

I have, like, zero tech knowledge, and have no idea how to actually do it, but I have heard that it's possible to block AI crawlers from being able to scrape your site for content.

It's only possible (or at least practical) to block "well-behaved" web crawlers — the ones that provide a distinct user-agent string that you add to your robots.txt file (or .htaccess file, or both). Most of the bandwidth-killer AI bots these days give you user-agent strings that look like regular browsers that people use like Chrome, Edge, Firefox, etc. The real problem, as is the case with everything else, is people, money, and the lack of credible international legal standards.

That said, it looks like AM is using a "funnel chute" approach to blocking in the robots.txt file, which you can do with data-driven websites like this, where he's disallowing everything on all but a few PHP script pages but allowing everything on the most-viewed pages, such as thechart.php. (I could easily be wrong about that, though...) Maybe under these circumstances, he could try blocking the well-behaved crawlers from the entire site by specifying them with a lengthy block-list (like this one), but the problem there is that those lists have to be kept up pretty regularly, and there are a lot of AI crawlers. And since they don't work on the ill-behaved bots, you're essentially punishing the well-behaved AI bots for being well-behaved.

albummaster

Janitor

Gender: Male

Site Admin

Thanks both for feedback, multiple layers to this (some of them more effective than others) and can't really reveal much here suffice to say 60,000% increase in Chinese traffic YoY, also massive increases in Singapore & Brazilian traffic etc (even US traffic artificially inflated). Main metric I personally use is active users to measure how busy the site is (number of people doing stuff on the site). Yes, BEA is used to crawlers, which were previously mostly search engines which were far better behaved (mostly), but at least they drove traffic to the site, so there was some kind of mutual benefit. Finding it incredibly frustrating, especially when spending so much time maintaining the site at my own expense and someone runs off with the content breaking stuff in their wake (at last six small outages over the past few weeks caused by AI crawlers).

Repo

BeA Sunflower

Location: Forest Park

albummaster wrote:

Thanks both for feedback, multiple layers to this (some of them more effective than others) and can't really reveal much here suffice to say 60,000% increase in Chinese traffic YoY, also massive increases in Singapore & Brazilian traffic etc (even US traffic artificially inflated). Main metric I personally use is active users to measure how busy the site is (number of people doing stuff on the site). Yes, BEA is used to crawlers, which were previously mostly search engines which were far better behaved (mostly), but at least they drove traffic to the site, so there was some kind of mutual benefit. Finding it incredibly frustrating, especially when spending so much time maintaining the site at my own expense and someone runs off with the content breaking stuff in their wake (at last six small outages over the past few weeks caused by AI crawlers).

I would be soooo angry & frustrated if I were you! Your calm & poise in the face of MASSIVE THEFT is inspiring. So sad that this is allowed to happen. It's fucking outrageous & infuriating.

🤬 😨

AAL2014

Fuck AI. Thanks for your work, sir!
_________________
Attention all planets of the solar federation: We have assumed control.