hugo/docs/content/extras/robots-txt.md

lastmod	date	menu	next	prev	title
2016-01-06	2015-12-08	main parent extras	/extras/datafiles	/extras/crossreferences	Custom robots.txt

Hugo can generated a customized robots.txt in the [same way as any other templates]({{< ref "templates/go-templates.md" >}}).

To enable it, just set enableRobotsTXT option to true in the [configuration file]({{< ref "overview/configuration.md" >}}). By default, it generates a robots.txt, which allows everything, with the following content:

User-agent: *

Hugo will use the template robots.txt according to the following list in descending precedence:

• /layouts/robots.txt
• /themes/THEME/layout/robots.txt

An example of a robots.txt layout is:

User-agent: *

{{range .Data.Pages}}
Disallow: {{.RelPermalink}}{{end}}

This template disallows and all the pages of the site creating one Disallow entry for each one.