overleaf/services/web/public/robots.txt

# robots.txt for https://www.sharelatex.com/

User-agent: *
Disallow: /project/*
Disallow: /recurly.com
Allow: /

User-Agent: AhrefsBot
Disallow: /

User-Agent: XoviBot
Disallow: /

User-Agent: RankSonicBot
Disallow: /

User-Agent: SMTBot
Disallow: /
Intial open source comment 2014-02-12 10:23:40 +00:00			`# robots.txt for https://www.sharelatex.com/`

			`User-agent: *`
			`Disallow: /project/*`
			`Disallow: /recurly.com`
added some robots to the blocked list 2015-09-22 13:16:58 +00:00			`Allow: /`

			`User-Agent: AhrefsBot`
			`Disallow: /`

			`User-Agent: XoviBot`
			`Disallow: /`

			`User-Agent: RankSonicBot`
			`Disallow: /`

			`User-Agent: SMTBot`
			`Disallow: /`