github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-12-27 06:22:12 +00:00
Code Issues Projects Releases Packages Wiki Activity

added some robots to the blocked list

Browse source
This commit is contained in:
Henry Oswald 2015-09-22 14:16:58 +01:00
parent 30a63d1e42
commit 27c02f5ef5
1 changed files with 13 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
services/web/public/robots.txt
View file

@ -3,3 +3,16 @@
User-agent: *
Disallow: /project/*
Disallow: /recurly.com
Allow: /
User-Agent: AhrefsBot
Disallow: /
User-Agent: XoviBot
Disallow: /
User-Agent: RankSonicBot
Disallow: /
User-Agent: SMTBot
Disallow: /