edsu/babble
1
0
Fork 0
forked from zesterer/babble
Code Pull requests Activity

Compare commits

base: edsu:main
Branches Tags
edsu:main
edsu:deny-robots
zesterer:main
..
compare: zesterer:main
Branches Tags
zesterer:main
edsu:main
edsu:deny-robots

2 commits
main ... main

Author SHA1 Message Date
Joshua Barretto
a86d7720c7 Merge pull request 'Deny all robots.txt respecting robots' (#1) from edsu/babble:deny-robots into main 
Reviewed-on: zesterer/babble#1
 2025-05-26 16:02:55 +02:00
Ed Summers
3ec4cd8595
Deny all robots 
Currently the robots.txt is set up to allow complete access by robots.
This means that well meaning bots that actually respect a sites wishes
with regards to crawling will be invited into the maze.

I think it makes more sense to tell all robots to go away, and if the
robot just blindly ignores this it will get lost in the babble tarpit.

Given enough babble instances this means that over time bot creators
will write LLM scraping bots that respect robots.txt so that they don't
incur the cost to their compute, bandwidth, and ultimately the quality
of their model.

```
To exclude all robots from the entire server

User-agent: *
Disallow: /

To allow all robots complete access

User-agent: *
Disallow:
```

via https://www.robotstxt.org/robotstxt.html
 2025-05-21 11:45:46 -04:00
1 changed files with 1 additions and 1 deletions
Show stats Expand all files Collapse all files

2
src/robots.txt
View file

@ -1,2 +1,2 @@
User-agent: * User-agent: *
Disallow: Disallow: /