hugo/content/en/templates/robots.md
Bjørn Erik Pedersen 07b8d9466d Squashed 'docs/' changes from 9cece6640..fb551cc75
fb551cc75 Update index.md
7af894857 Update index.md
d235753ea Hugo 0.82.1
e03e72deb Merge branch 'temp0821'
e62648961 Merge branch 'release-0.82.1'
e1ab0f6eb releaser: Add release notes to /docs for release of 0.82.1
5d354c38d Replaced ``` code blocks with Code Toggler
c9d065c20 Remove duplicate YAML keys (#1420)
8ae31e701 Add webp image encoding support
848f2af26 Update internal.md (#1407)
c103a86a4 Fix `ref` shortcode example output (#1409)
9f8ba56dc Remove leading dot from where function KEY (#1419)
363251a51 Improve presentation of template lookup order (#1382)
b73da986d Improve description of Page Resources (#1381)
4e0bb96d5 Rework robots.txt page (#1405)
edf893e6f Update migrations.md (#1412)
450f1580b Add link to `site` function doc (#1417)
cfffa6e6f Added one extension to the list (#1414)
05f1665a0 Update theme
5de0b1c6a Update theme
250e20552 Add hugo.IsExtended
dea5e1fd7 Fix typo on merge function page (#1408)
1bbed2cf3 Update configuration.md
be0b64a46 Omit ISO
cbb5b8367 Fix `dateFormat` documentation
698f15466 Regenerate the docshelper
f9a8a7cb6 Update multilingual.md
a22dc6267 Fix grammar (#1398)
eb98b0997 Fix pretty URL example (#1397)
f4c4153dc Mention date var complementation in post scheduling (#1396)
17fae284c Fix resources.ExecuteAsTemplate argument order (#1394)
97e2c2abb Use code-toggle shortcode in `multilingual.md` (#1388)
3a84929bb Harmonize capitalization (#1393)
17f15daa6 fix file naming used in example (#1392)
5d97b6a18 Add slice syntax to sections permalinks config
00665b97b Improve description of `site.md`
edcf5e3fc Fix example in `merge.md`
f275ab778 Update postprocess.md
9593e3991 Fix file name
59bd9656f Update postprocess.md
1172fb6d0 Update to theNewDynamic repository (#1263)
f5b5c1d2c Update Hugo container image
4f2e92f2a Adapt anchorize.md to Goldmark
98aa19073 Directly link to `highlight` shortcode (#1384)
4c75c2422 Fix header level
f15c06f23 markdownify: add note about render-hooks and .RenderString (#1281)
69c82eb68 Remove Blackfriday reference from shortcode desc (#1380)
36de478df Update description of ignoreFiles config setting (#1377)
6337699d8 Remove "Authors" page from documentation (#1371)
35e73ca90 fix indent in example (#1372)
d3f01f19a Remove opening body tag from header example (#1376)
341a5a7d8 Update index.md
c9bfdbee6 Release 0.82.0
119644949 releaser: Add release notes to /docs for release of 0.82.0
32efaed78 docs: Regenerate docs helper
dea5449a2 docs: Regen CLI docs
eeab18fce Merge commit '81689af79901f0cdaff765cda6322dd4a9a7ccb3'
d508a1259 Attributes for code fences should be placed after the lang indicator only
c80905cef deps: Update to esbuild v0.9.0
95350eb79 Add support for Google Analytics v4
02d36f9bc Allow markdown attribute lists to be used in title render hooks
7df220a64 Merge commit '9d31f650da964a52f05fc27b7fb99cf3e09778cf'
d80bf61b7 Fixes #7698.

git-subtree-dir: docs
git-subtree-split: fb551cc750faa83a1493b0e0d0898cd98ab74465
2021-04-20 20:21:45 +02:00

1.9 KiB

title linktitle description date publishdate lastmod categories keywords menu weight sections_weight draft aliases toc
Robots.txt File Robots.txt Hugo can generate a customized robots.txt in the same way as any other template. 2017-02-01 2017-02-01 2017-02-01
templates
robots
search engines
docs
parent weight
templates 165
165 165 false
/extras/robots-txt/
false

To generate a robots.txt file from a template, change the site configuration:

{{< code-toggle file="config">}} enableRobotsTXT = true {{< /code-toggle >}}

By default, Hugo generates robots.txt using an internal template.

User-agent: *

Search engines that honor the Robots Exclusion Protocol will interpret this as permission to crawl everything on the site.

Robots.txt Template Lookup Order

You may overwrite the internal template with a custom template. Hugo selects the template using this lookup order:

  1. /layouts/robots.txt
  2. /themes/<THEME>/layouts/robots.txt

Robots.txt Template Example

{{< code file="layouts/robots.txt" download="robots.txt" >}} User-agent: * {{ range .Pages }} Disallow: {{ .RelPermalink }} {{ end }} {{< /code >}}

This template creates a robots.txt file with a Disallow directive for each page on the site. Search engines that honor the Robots Exclusion Protocol will not crawl any page on the site.

{{% note %}} To create a robots.txt file without using a template:

  1. Set enableRobotsTXT to false in the site configuration.
  2. Create a robots.txt file in the static directory.

Remember that Hugo copies everything in the static directory to the root of publishDir (typically public) when you build your site.

{{% /note %}}