zesterer/babble
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Compare commits

base: zesterer:a77eb52c564197b03a6b03a49fdddde3888cac39
Branches Tags
zesterer:main
..
compare: zesterer:a86d7720c71d0d4596f5b5d7022dd267a5e1487e
Branches Tags
zesterer:main

2 commits
a77eb52c56 ... a86d7720c7

Author SHA1 Message Date
Joshua Barretto
a86d7720c7 Merge pull request 'Deny all robots.txt respecting robots' (#1) from edsu/babble:deny-robots into main 
Reviewed-on: #1
 2025-05-26 16:02:55 +02:00
Ed Summers
3ec4cd8595
Deny all robots 
Currently the robots.txt is set up to allow complete access by robots.
This means that well meaning bots that actually respect a sites wishes
with regards to crawling will be invited into the maze.

I think it makes more sense to tell all robots to go away, and if the
robot just blindly ignores this it will get lost in the babble tarpit.

Given enough babble instances this means that over time bot creators
will write LLM scraping bots that respect robots.txt so that they don't
incur the cost to their compute, bandwidth, and ultimately the quality
of their model.

```
To exclude all robots from the entire server

User-agent: *
Disallow: /

To allow all robots complete access

User-agent: *
Disallow:
```

via https://www.robotstxt.org/robotstxt.html
 2025-05-21 11:45:46 -04:00
1 changed files with 1 additions and 1 deletions
Show stats Expand all files Collapse all files

2
src/robots.txt
View file

@ -1,2 +1,2 @@
User-agent: * User-agent: *
Disallow: Disallow: /