erik/mastodon
1
0
Fork 0
mirror of https://github.com/mastodon/mastodon.git synced 2024-08-20 21:08:15 -07:00
Code Issues Projects Releases Wiki Activity

Update robots.txt

Browse source
This commit is contained in:
Shlee 2024-08-01 20:41:10 +09:30 committed by GitHub
parent b4c026fe4f
commit cc7e07f12f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 2 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
public/robots.txt
View file

@ -8,7 +8,7 @@ Disallow: /interact/
# AI Bots - Crawlers/scrapers and data harvesters. Blocking these should not negatively effect the end user.
###############################################################################################################################
# OpenAI Scraper/Crawler/Assistant - https://platform.openai.com/docs/bots
# OpenAI Scraper/Crawler/Assistant https://platform.openai.com/docs/bots
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
@ -22,7 +22,7 @@ User-agent: Applebot
# Apple AI models https://support.apple.com/en-au/119829
User-agent: Applebot-Extended
# https://commoncrawl.org/ccbot
# Common Crawl https://commoncrawl.org/ccbot
User-agent: CCBot
# Facebook AI models https://developers.facebook.com/docs/sharing/bot/
@ -33,8 +33,6 @@ User-agent: Meta-ExternalAgent
# Googles Gemini Apps and Vertex AI https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
User-agent: Google-Extended
###############################################################################################################################
# https://docs.perplexity.ai/docs/perplexity-bot
User-agent: PerplexityBot