erik/mastodon
1
0
Fork 0
mirror of https://github.com/mastodon/mastodon.git synced 2024-08-20 21:08:15 -07:00
Code Issues Projects Releases Wiki Activity

Update robots.txt

Browse source
This commit is contained in:
Shlee 2024-08-01 11:58:24 +09:30 committed by GitHub
parent a50c8e951f
commit 10c5cd4c99
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 74 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

77
public/robots.txt
View file

@ -1,8 +1,79 @@
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /media_proxy/
Disallow: /interact/
## AI Bots - Research from darkvisitors.com
# AI Search Crawler
User-agent: Amazonbot
Disallow: /
# AI Search Crawler
User-agent: Applebot
Disallow: /
# AI Data Scraper
User-agent: Applebot-Extended
Disallow: /
# AI Data Scraper
User-agent: Bytespider
Disallow: /
# AI Data Scraper
User-agent: CCBot
Disallow: /
# AI Data Scraper
User-agent: GPTBot
Disallow: /
# AI Assistant
User-agent: ChatGPT-User
Disallow: /
# AI Data Scraper
User-agent: ClaudeBot
Disallow: /
# AI Data Scraper
User-agent: Diffbot
Disallow: /
# AI Data Scraper
User-agent: FacebookBot
Disallow: /
# AI Data Scraper
User-agent: Google-Extended
Disallow: /
# AI Data Scraper
User-agent: Meta-ExternalAgent
Disallow: /
# AI Assistant
User-agent: Meta-ExternalFetcher
Disallow: /
# AI Search Crawler
User-agent: OAI-SearchBot
Disallow: /
# AI Data Scraper
User-agent: omgili
Disallow: /
# AI Search Crawler
User-agent: PerplexityBot
Disallow: /
# AI Data Scraper
User-agent: Timpibot
Disallow: /
# AI Search Crawler
User-agent: YouBot
Disallow: /