hugo/docs/content/extras/robots-txt.md

lastmod	date	menu	next	prev	title	weight
2016-01-06	2015-12-08	main parent extras	/community/mailing-list	/extras/urls	Custom robots.txt	120

Hugo can generated a customized robots.txt in the [same way as any other templates]({{< ref "templates/go-templates.md" >}}).

By default, it generates a robots.txt, which allows everything, with the following content:

User-agent: *

To disable it, just set disableRobotsTXT option to false in the [command line]({{< ref "commands/hugo.md" >}}) or [configuration file]({{< ref "overview/configuration.md" >}}).

Hugo will use the template robots.txt according to the following list in descending precedence:

• /layouts/robots.txt
• /themes/THEME/layout/robots.txt

An example of a robots.txt layout is:

User-agent: *

{{range .Data.Pages}}
Disallow: {{.RelPermalink}}{{end}}

This template disallows and all the pages of the site creating one Disallow entry for each one.