erik/mastodon
1
0
Fork 0
mirror of https://github.com/mastodon/mastodon.git synced 2024-08-20 21:08:15 -07:00
Code Issues Projects Releases Wiki Activity

Update robots.txt

Browse source
This commit is contained in:
Shlee 2024-08-02 00:36:36 +09:30 committed by GitHub
parent ade04e630b
commit 776676cec4
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 1 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
public/robots.txt
View file

@ -35,6 +35,7 @@ User-agent: Applebot-Extended
Disallow: / Disallow: /
# Common Crawl https://commoncrawl.org/ccbot # Common Crawl https://commoncrawl.org/ccbot
# "democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone."
User-agent: CCBot User-agent: CCBot
Disallow: / Disallow: /