From 3ec4cd859592050a762c3878a0c5e60d74458a06 Mon Sep 17 00:00:00 2001 From: Ed Summers Date: Wed, 21 May 2025 11:45:46 -0400 Subject: [PATCH] Deny all robots Currently the robots.txt is set up to allow complete access by robots. This means that well meaning bots that actually respect a sites wishes with regards to crawling will be invited into the maze. I think it makes more sense to tell all robots to go away, and if the robot just blindly ignores this it will get lost in the babble tarpit. Given enough babble instances this means that over time bot creators will write LLM scraping bots that respect robots.txt so that they don't incur the cost to their compute, bandwidth, and ultimately the quality of their model. ``` To exclude all robots from the entire server User-agent: * Disallow: / To allow all robots complete access User-agent: * Disallow: ``` via https://www.robotstxt.org/robotstxt.html --- src/robots.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/robots.txt b/src/robots.txt index eb05362..1f53798 100644 --- a/src/robots.txt +++ b/src/robots.txt @@ -1,2 +1,2 @@ User-agent: * -Disallow: +Disallow: /