hugo/docs/content/en/templates/robots.md

---
title: Robots.txt File
linktitle: Robots.txt
description: Hugo can generate a customized robots.txt in the same way as any other template.
date: 2017-02-01
publishdate: 2017-02-01
lastmod: 2017-02-01
categories: [templates]
keywords: [robots,search engines]
menu:
  docs:
    parent: "templates"
    weight: 165
weight: 165
sections_weight: 165
draft: false
aliases: [/extras/robots-txt/]
toc: false
---

To create your robots.txt as a template, first set the `enableRobotsTXT` value to `true` in your [configuration file][config]. By default, this option generates a robots.txt with the following content, which tells search engines that they are allowed to crawl everything:

```
User-agent: *
```

## Robots.txt Template Lookup Order

The [lookup order][lookup] for the `robots.txt` template is as follows:

* `/layouts/robots.txt`
* `/themes/<THEME>/layouts/robots.txt`

{{% note %}}
If you do not want Hugo to create a default `robots.txt` or leverage the `robots.txt` template, you can hand code your own and place the file in `static`. Remember that everything in the [static directory](/getting-started/directory-structure/) is copied over as-is when Hugo builds your site.
{{% /note %}}

## Robots.txt Template Example

The following is an example `robots.txt` layout:

{{< code file="layouts/robots.txt" download="robots.txt" >}}
User-agent: *

{{range .Pages}}
Disallow: {{.RelPermalink}}
{{end}}
{{< /code >}}

This template disallows all the pages of the site by creating one `Disallow` entry for each page.

[config]: /getting-started/configuration/
[lookup]: /templates/lookup-order/
[robots]: http://www.robotstxt.org/
Squashed 'docs/' content from commit f887bd7b git-subtree-dir: docs git-subtree-split: f887bd7b4e3e7c7e76cd63951e5b0d37d8fe0ac7 2017-08-10 15:18:22 +00:00			`---`
			`title: Robots.txt File`
			`linktitle: Robots.txt`
			`description: Hugo can generate a customized robots.txt in the same way as any other template.`
			`date: 2017-02-01`
			`publishdate: 2017-02-01`
			`lastmod: 2017-02-01`
			`categories: [templates]`
Squashed 'docs/' changes from 44fe0285..32356e4e 32356e4e Fix typo in header of shortcode-templates.md c8f1a2d2 Correct code example for index template function bfa6a55d Escape code fencing ff8b2f99 Fix typos in deployment with wercker tutorial 557c36e8 theme: Merge commit '7fbb4bed25001182bfeb91f79db0f0c1936582ee' 7fbb4bed Squashed 'themes/gohugoioTheme/' changes from 7dd8a302..ca53082d ce31cee0 Add "See Also" config 158cee1b Make the tags into keywords 61600be6 Add a note to the related section 49edb5a2 Relase 0.27.1 c9bbc001 releaser: Add release notes to /docs for release of 0.27.1 213c6c3b Add bugs poster 8b4590cd Add KeyCDN integration tutorial 2b277859 Add tutorial videos to several docs pages 950fef1f Update roadmap to link to the correct milestones page 496f5bf6 Rename relnotes d6f9378d Bump Netlify versions to 0.27 087fde7f Update 0.27 release notes 603f94ae docs: Document Related Content 3790f6a3 releaser: Bump versions for release of 0.27 0948868c releaser: Add release notes to /docs for release of 0.27 git-subtree-dir: docs git-subtree-split: 32356e4eabe357ae914f4d1d59e8ae31ce936723 2017-09-21 17:03:00 +00:00			`keywords: [robots,search engines]`
Squashed 'docs/' content from commit f887bd7b git-subtree-dir: docs git-subtree-split: f887bd7b4e3e7c7e76cd63951e5b0d37d8fe0ac7 2017-08-10 15:18:22 +00:00			`menu:`
			`docs:`
			`parent: "templates"`
			`weight: 165`
			`weight: 165`
			`sections_weight: 165`
			`draft: false`
			`aliases: [/extras/robots-txt/]`
			`toc: false`
			`---`

			To create your robots.txt as a template, first set the `enableRobotsTXT` value to `true` in your [configuration file][config]. By default, this option generates a robots.txt with the following content, which tells search engines that they are allowed to crawl everything:

			```
			`User-agent: *`
			```

			`## Robots.txt Template Lookup Order`

			The [lookup order][lookup] for the `robots.txt` template is as follows:

			* `/layouts/robots.txt`
Squashed 'docs/' changes from 000ab7c42..4628b9ec2 4628b9ec2 commands: Regen CLI doc 2525f2ed0 data: Regenerate docs helper 6f5a0eb19 Add Hugo 0.30 poster image 72c3fac9e Merge branch 'chroma-next2' into next 364973d3f Fix typo in syntax highlighting. ce10cc02e Update Chroma highlighting docs 9dcc4d4dd Update robots.md 1e64cb483 Rename title of cross references' page d6dfbbc51 Add warning about MMark and TOCs e8d259d32 Fix link to subsection in page 6adead19d Merge commit '040d8d2833c26c53cf9f0e035910821ed50e3863' 040d8d283 Squashed 'themes/gohugoioTheme/' changes from cdaa89c8..6b632895 bde95d890 Add Atlas starter kit fc40d078d Remove page arg from examples of relref shortcode c578620b5 Remove page arg from examples of ref shortcode ee81931a4 Remove delimiters in YAML and TOML config examples 62d7b269f Clarify that .Lastmod automatically uses .GitInfo.AuthorDate (#226) git-subtree-dir: docs git-subtree-split: 4628b9ec2c52df4de673a4d6b9621a65d8e8f5a4 2017-10-15 08:20:55 +00:00			* `/themes/<THEME>/layouts/robots.txt`
Squashed 'docs/' content from commit f887bd7b git-subtree-dir: docs git-subtree-split: f887bd7b4e3e7c7e76cd63951e5b0d37d8fe0ac7 2017-08-10 15:18:22 +00:00
			`{{% note %}}`
			If you do not want Hugo to create a default `robots.txt` or leverage the `robots.txt` template, you can hand code your own and place the file in `static`. Remember that everything in the [static directory](/getting-started/directory-structure/) is copied over as-is when Hugo builds your site.
			`{{% /note %}}`

			`## Robots.txt Template Example`

			The following is an example `robots.txt` layout:

			`{{< code file="layouts/robots.txt" download="robots.txt" >}}`
			`User-agent: *`

Squashed 'docs/' changes from f59b3ab06..cfe89ecbc cfe89ecbc Set all *.png files under static/images to mode 0644 d847ccd8b More spelling error fixes bf1405d92 Correction: .Pages on homepage is same as .Site.RegularPages 7efa41ff5 Merge commit 'd137efe0222269e09b427325176f0044558e3cc5' d137efe02 Squashed 'themes/gohugoioTheme/' changes from f31a3dc8..66249819 50df8bfb8 Replace .Data.Pages with .Pages 85d3712d0 Mention the newScratch template func 1c234db1b Release 0.44 a51cb3e36 Merge branch 'temp44' ec424a5e2 releaser: Prepare repository for 0.45-DEV 425e85574 releaser: Add release notes to /docs for release of 0.44 e6f9c65a2 releaser: Bump versions for release of 0.44 45548a5c9 Arch Linux: Partial upgrades are unsupported c0430f8f9 Update MenuEntry properties documentation 79109b685 Refresh the documentation on staticDir config parameter 80ed4592b Remove duplicate links; also sort them alphabetically f00a4e232 HTTPS links to third-party sites 2f6f682ba Fixed a typo 4be105202 add openbsd howto install 39808f50a Fix typo b55c0f3a0 Fix URL to hugotest 2fb157237 Release Hugo 0.43 e8af6a970 Merge branch 'temp43' 563dd4173 Adjust release notes 50aed2e52 releaser: Prepare repository for 0.44-DEV 363b363ff releaser: Add release notes to /docs for release of 0.43 93bfbef62 releaser: Bump versions for release of 0.43 893a27d33 Fix typos fe5908257 Celebrate: a few typo and grammar fixes bb20442df Merge commit '98293eaa1570b5aff4452021c8b6d6c8560b3f06' ef3d63936 Add a newScratch template func bf100a3cd Merge branch 'release-0.42.2' f1133c544 releaser: Prepare repository for 0.43-DEV 2ce058057 releaser: Add release notes to /docs for release of 0.42.2 6073927c6 releaser: Bump versions for release of 0.42.2 3bafddd52 releaser: Prepare repository for 0.43-DEV 8b9f805bd releaser: Add release notes to /docs for release of 0.42.1 efb6638ba releaser: Bump versions for release of 0.42.1 1e74ffbe0 releaser: Prepare repository for 0.43-DEV 2d2d0f8ab releaser: Add release notes to /docs for release of 0.42 17f685813 releaser: Bump versions for release of 0.42 67ef1a0e1 docs: Update theme documentation d74874e1e docs: Remove some files now moved 771d50cf3 Merge commit 'b239595af5a9fc1fc9a1ccc666c3ab06ccc32f04' f9b5fd2d2 tplimpl: Remove speakerdeck shortcode 523206579 tpl/strings: strings.RuneCount b389efc88 tpl: Add strings.Repeat a41ad0254 Add a BlackFriday option for rel="noreferrer" on external links a6e279362 Add a BlackFriday option for rel="nofollow" on external links 7e2c9846e releaser: Prepare repository for 0.42-DEV b4f81c38a releaser: Add release notes to /docs for release of 0.41 55667b3a1 releaser: Bump versions for release of 0.41 afceb02d3 docs: Document the GDPR Privacy Config 8b8289027 Merge commit 'd2b1030060d3c91d5f9ffa3456418da16bd74f1d' be04a8886 Merge branch 'release-0.40.3' 33ca0b8da releaser: Prepare repository for 0.41-DEV 8c9a5a850 releaser: Add release notes to /docs for release of 0.40.3 85845a4fd releaser: Bump versions for release of 0.40.3 9af79e531 Merge commit '83bef6955e014d40c0f00db9cebe09113154e999' c4200dd0e Fix typo 8f8323336 releaser: Prepare repository for 0.41-DEV 9fab87505 releaser: Add release notes to /docs for release of 0.40.2 83648c666 releaser: Bump versions for release of 0.40.2 e6434d104 releaser: Prepare repository for 0.41-DEV 9389b2973 releaser: Add release notes to /docs for release of 0.40.1 32979e28a releaser: Bump versions for release of 0.40.1 196174a76 releaser: Prepare repository for 0.41-DEV 490a997bb releaser: Add release notes to /docs for release of 0.40 018aa6471 releaser: Bump versions for release of 0.40 92221934c Merge commit 'a215abf70e018f4bf40d6c09d8bd148d8684b33d' 0a58ef56e Merge commit 'd2ec1a06df8ab6b17ad05cb008d5701b40327d47' 312ab58c6 Improve .Get docs e43e8cee3 .Get function: fix syntax signature bb16150bc releaser: Prepare repository for 0.40-DEV 8cc80fed9 releaser: Add release notes to /docs for release of 0.39 f7aa0d1c5 releaser: Bump versions for release of 0.39 42e48706a Merge commit '047c4188dfc854f658d16f1e4a9501f9c97a31c7' 76acab68b releaser: Prepare repository for 0.39-DEV d270600b0 releaser: Add release notes to /docs for release of 0.38.2 97be36992 releaser: Bump versions for release of 0.38.2 cd29b8b79 releaser: Prepare repository for 0.39-DEV 9983a4bdf releaser: Add release notes to /docs for release of 0.38.1 7774afd01 releaser: Bump versions for release of 0.38.1 b89157f20 releaser: Prepare repository for 0.39-DEV af4d0bf62 releaser: Add release notes to /docs for release of 0.38 ca98e7d4f releaser: Bump versions for release of 0.38 8b773833a Merge commit 'ed8bf081fdbf336e026517b7e1b123c039014ab5' 24202bb71 docs: Generate docshelper data 43d15975d Add .Site.IsServer 6f99d3d65 Merge commit '0a23baa6a90901f772c234107c4f12c16c76f4aa' 17487ccdd hugolib: Add Reset method to delete key from Scratch ccbf7a05c docs: Add docs for lang.Merge 5ad46d78e Merge commit '3886fc1fef6ac19d58b9ba1bb642d0c6c9a54031' ac403c280 docs: Add docs on the new front matter configuration 6066c60e6 Merge commit 'c0290655825e7bb36e13fb39f89d85b392cf1adc' ee605c7ae releaser: Prepare repository for 0.38-DEV 31a4de966 releaser: Add release notes to /docs for release of 0.37.1 9dd3dafe1 releaser: Bump versions for release of 0.37.1 141750aec releaser: Prepare repository for 0.38-DEV c9bb39bc3 releaser: Add release notes to /docs for release of 0.37 c7009f064 releaser: Bump versions for release of 0.37 9f727d916 Merge commit '900b5f6cfe5a377ef369d26cd700201be4cf6b06' 87fd97853 Merge commit '374d184e6747678364fd61f5faf328ec9205eb6b' 244deafc2 Fix typos in development contribution doc 608c80f72 Merge branch 'release-0.36.1' 0f8ffbd5c releaser: Prepare repository for 0.37-DEV 5028ee849 releaser: Add release notes to /docs for release of 0.36.1 9ccc7cb11 releaser: Bump versions for release of 0.36.1 3db31be07 Merge commit '9cc9bab46288d8d5f9fda7009c5f746258cec1b4' 9468246c5 Add "target" and "rel" parameters to figure shortcode 6d7fa89f9 releaser: Prepare repository for 0.37-DEV 1aec7ad81 releaser: Add release notes to /docs for release of 0.36 b522da389 releaser: Bump versions for release of 0.36 bb78bea13 docs: Add documentation for smart cropping etc. 930f16cae Merge commit 'c305e44f5f081e4436195923a4593e396f07cd49' 0e4c80ab9 releaser: Prepare repository for 0.36-DEV e0d4c6bcd releaser: Add release notes to /docs for release of 0.35 f4ac67eb3 releaser: Bump versions for release of 0.35 229a50b1f docs: Regenerate CLI docs 72958eb9c Merge commit '337d0c5f516ee085205e8abefdb7f87e6d33ca05' d998ca4c0 command: Remove undraft command 34a0052ef docs: Re-generate CLI docs fa04ff331 releaser: Prepare repository for 0.35-DEV 55e595720 releaser: Add release notes to /docs for release of 0.34 bf4d0e8fb releaser: Bump versions for release of 0.34 07673370e releaser: Prepare repository for 0.34-DEV 921ea8920 releaser: Add release notes to /docs for release of 0.33 3307f61bd releaser: Bump versions for release of 0.33 792a4410d Merge commit '3cf4300097610bb8b5bd0686d96d1df5db641895' 6d8a9f657 releaser: Prepare repository for 0.33-DEV e3afccf6f releaser: Add release notes to /docs for release of 0.32.4 7f29cb196 releaser: Bump versions for release of 0.32.4 baa364436 releaser: Prepare repository for 0.33-DEV 60c775166 releaser: Add release notes to /docs for release of 0.32.3 9cbab1803 releaser: Bump versions for release of 0.32.3 51823efee releaser: Prepare repository for 0.33-DEV 71e03684c releaser: Add release notes to /docs for release of 0.32.2 e18bf56fb releaser: Bump versions for release of 0.32.2 54aa93927 Merge commit 'eb738cd35cca1ffc68c5ed688dbe2a19108e8761' 5e8123994 releaser: Prepare repository for 0.33-DEV a55b78124 releaser: Add release notes to /docs for release of 0.32.1 7de53ff41 releaser: Bump versions for release of 0.32.1 e6e58785d releaser: Prepare repository for 0.33-DEV 58cf5009d releaser: Add release notes to /docs for release of 0.32 031719821 Merge commit 'f3cd083961f36dc96d05e98aaf67f650102bc757' b9183604b Add Pandoc support, refactor external helpers 3f9b9474c releaser: Prepare repository for 0.32-DEV 11e4ce514 releaser: Add release notes to /docs for release of 0.31.1 aebb1504b releaser: Bump versions for release of 0.31.1 d70b54d4a releaser: Prepare repository for 0.32-DEV cdc10468e releaser: Add release notes to /docs for release of 0.31 38bd41cab releaser: Bump versions for release of 0.31 1c581a11d Merge commit '30c0d485eaff6d70df1be0353911ddca485d52bf' b88e46d02 Merge commit '05e42bc643f1840ed2ad9c2eff82a269d1381683' a45ab174e Handle Taxonomy permalinks 9f27354f6 Add support for height argument to figure shortcode acc8e49c1 releaser: Prepare repository for 0.31-DEV 977266b19 releaser: Add release notes to /docs for release of 0.30.2 31b672844 releaser: Bump versions for release of 0.30.2 b8f6b72a8 Merge commit '325009c3fd4ac90021897b7e3e025c14e70ce162' c6e3dae71 releaser: Prepare repository for 0.31-DEV 2a3c7c7d9 releaser: Add release notes to /docs for release of 0.30.1 4c014117a releaser: Bump versions for release of 0.30.1 1da61da06 releaser: Prepare repository for 0.31-DEV 24864a46b releaser: Add release notes to /docs for release of 0.30 3011aa44a releaser: Bump versions for release of 0.30 bcbf3237b Merge commit 'ecf5e081b5540e69f4af330233f39a07baf53846' ab68b99ed Merge commit 'dae5a7c61cceeb0de59f2d755f63e453f71dd9b2' 0ae435725 tpl: Add errorf template function ac3fb3808 Change SummaryLength to be configurable (#3924) 3c0e4fa70 tpl: Add os.fileExists template function 9290f3983 Merge commit '9d68f695e782c6a83c77aff13317c7a22c694c98' 394e8b3f6 tpl: Add float template function c0c48c2ea releaser: Prepare repository for 0.30-DEV 0806d910b releaser: Add release notes to /docs for release of 0.29 0d1101836 releaser: Bump versions for release of 0.29 5d92b552b releaser: Prepare repository for 0.29-DEV 1ef521182 releaser: Add release notes to /docs for release of 0.28 c5441cff1 releaser: Bump versions for release of 0.28 46c2786f1 Merge commit '61c27b58b353c73772aae572c7d822fdfdf7791b' 6fed4008f Merge commit '30694a133a88d5f76a51d0372646e10cbeca7691' 595752e2f Merge commit '7a89dce53bfbd67a17442a8f9be8fa895fc4f9b1' af14cae0c Merge commit 'ba45da9d03056447e4873de13d4e0f8d658a769b' d0bb30963 releaser: Prepare repository for 0.28-DEV 469bf26aa releaser: Add release notes to /docs for release of 0.27.1 06ea00c12 releaser: Bump versions for release of 0.27.1 0668af58e releaser: Prepare repository for 0.28-DEV 4ce00c84d releaser: Add release notes to /docs for release of 0.27 68f318d0b releaser: Bump versions for release of 0.27 509ad6cc0 docs: Merge commit '1b4319be62ba071f79e90ef32dbe92eb893429f7' c625ae1ce docs: Document Related Content ecd5c24f5 docs: Merge commit '7d63a23b0c68d9cd7c7c09c2755619237bc03485' b97b84f9d Update docs versiona and README 2238f7a4a Merge commit 'ec4e6f9df2ab9ffdc62a3f59675369096e0d3f77' as 'docs' git-subtree-dir: docs git-subtree-split: cfe89ecbc2288a7270c23dbcc179733c8d978c4c 2018-07-18 09:04:57 +00:00			`{{range .Pages}}`
Squashed 'docs/' content from commit f887bd7b git-subtree-dir: docs git-subtree-split: f887bd7b4e3e7c7e76cd63951e5b0d37d8fe0ac7 2017-08-10 15:18:22 +00:00			`Disallow: {{.RelPermalink}}`
			`{{end}}`
			`{{< /code >}}`

			This template disallows all the pages of the site by creating one `Disallow` entry for each page.

			`[config]: /getting-started/configuration/`
			`[lookup]: /templates/lookup-order/`
Squashed 'docs/' changes from 000ab7c42..4628b9ec2 4628b9ec2 commands: Regen CLI doc 2525f2ed0 data: Regenerate docs helper 6f5a0eb19 Add Hugo 0.30 poster image 72c3fac9e Merge branch 'chroma-next2' into next 364973d3f Fix typo in syntax highlighting. ce10cc02e Update Chroma highlighting docs 9dcc4d4dd Update robots.md 1e64cb483 Rename title of cross references' page d6dfbbc51 Add warning about MMark and TOCs e8d259d32 Fix link to subsection in page 6adead19d Merge commit '040d8d2833c26c53cf9f0e035910821ed50e3863' 040d8d283 Squashed 'themes/gohugoioTheme/' changes from cdaa89c8..6b632895 bde95d890 Add Atlas starter kit fc40d078d Remove page arg from examples of relref shortcode c578620b5 Remove page arg from examples of ref shortcode ee81931a4 Remove delimiters in YAML and TOML config examples 62d7b269f Clarify that .Lastmod automatically uses .GitInfo.AuthorDate (#226) git-subtree-dir: docs git-subtree-split: 4628b9ec2c52df4de673a4d6b9621a65d8e8f5a4 2017-10-15 08:20:55 +00:00			`[robots]: http://www.robotstxt.org/`