mirror of
https://github.com/gohugoio/hugo.git
synced 2024-11-14 20:37:55 -05:00
a6e635ca7d
fcc3ed651 Remove some expired new-in a9c5981f5 Fix cascade example 82bb250fa Add some lines about permalinks tokens in front matter 328fe564e Remove some outdated new-in fb140b153 Hide showcase menu entry 42d9d1c79 Update image formats from which EXIF data can be extracted 09ad56b6e netlify: Hugo 0.130.0 1d503f846 Merge branch 'tempv0.130.0' e2458074d math: Add trigonometric functions and some angle helper functions 392afc8f9 Disable the showcase section for now 0300750f2 Improve example of image render hook 60a9306af Improve description of the .Site.RegularPages method 8d759175d Fix typos 55daa4554 Update XxHash.md 397c81cb7 Add namespace for hash functions 70fe8d2f0 netlify: Bump Hugo 0.129.0 5a9771aff Merge branch 'tempv0.129.0' f9146575b Fix typo e6e1fea49 Fix typo in Hugo docs | functions | partial 732d10ec4 source: Expose GitInfo Body 34c97e639 netlify: Hugo 0.128.2 3270587e9 Fix typo 727c5396e netlify: Hugo 0.128.1 80b6ae99c Update GitHub Pages workflow file example 027134102 Update GitHub Pages workflow file example 2600a8a2e Miscellaneous edits 3fdd5819b Update Build.md 7764005c3 Improve example of render hook directory structure 5e3941d82 Fix typos 748bf065f Restructure templates section fafbf6566 Update Defer.md 012162e0d Document changes to template functions in v0.128.0 0990ce35b quick-reference: Update emojis 6677a30ef Update Goldmark configuration documentation 4449d530d Document new pagination config 0af8be439 Update Defer.md 56348196d netlify: Hugo 0.128.0 d67b6d82e Update content/en/functions/templates/Defer.md 23d996b3d Update content/en/functions/templates/Defer.md 7f7fb2f27 Document templates.Defer 5ada1e9d5 Fix docs merge (remove shortcode) d27ee6156 Merge branch 'tempv0.128.0' 5d7317c84 Fix typo 7c18ee546 Update theme 83bfea63b Update theme b274b3238 Merge commit '8b9803425e63e1b1801f8d5d676e96368d706722' ff34a035a deploy: Add stripIndexHtml target option d9e964bdb markup/goldmark: Add the Hugo Goldmark Extras "delete" extension ac5bd16d2 deps: Upgrade github.com/alecthomas/chroma v2.13.0 => v2.14.0 25377171b config: Remove extraneous BuildConfig setting 0d2044f6d docs: Regen docshelper a2548dac9 markup/goldmark: Support extras extension 9d0c86ee8 commands: Add gen chromastyles --lineNumbersTableStyle flag git-subtree-dir: docs git-subtree-split: fcc3ed651a1b6431303c2f88f20fa38531c52b3d
60 lines
1.7 KiB
Markdown
60 lines
1.7 KiB
Markdown
---
|
|
title: robots.txt template
|
|
linkTitle: robots.txt templates
|
|
description: Hugo can generate a customized robots.txt in the same way as any other template.
|
|
categories: [templates]
|
|
keywords: []
|
|
menu:
|
|
docs:
|
|
parent: templates
|
|
weight: 170
|
|
weight: 170
|
|
aliases: [/extras/robots-txt/]
|
|
---
|
|
|
|
To generate a robots.txt file from a template, change the [site configuration]:
|
|
|
|
{{< code-toggle file=hugo >}}
|
|
enableRobotsTXT = true
|
|
{{< /code-toggle >}}
|
|
|
|
By default, Hugo generates robots.txt using an [embedded template].
|
|
|
|
[embedded template]: {{% eturl robots %}}
|
|
|
|
```text
|
|
User-agent: *
|
|
```
|
|
|
|
Search engines that honor the Robots Exclusion Protocol will interpret this as permission to crawl everything on the site.
|
|
|
|
## robots.txt template lookup order
|
|
|
|
You may overwrite the internal template with a custom template. Hugo selects the template using this lookup order:
|
|
|
|
1. `/layouts/robots.txt`
|
|
2. `/themes/<THEME>/layouts/robots.txt`
|
|
|
|
## robots.txt template example
|
|
|
|
{{< code file=layouts/robots.txt >}}
|
|
User-agent: *
|
|
{{ range .Pages }}
|
|
Disallow: {{ .RelPermalink }}
|
|
{{ end }}
|
|
{{< /code >}}
|
|
|
|
This template creates a robots.txt file with a `Disallow` directive for each page on the site. Search engines that honor the Robots Exclusion Protocol will not crawl any page on the site.
|
|
|
|
{{% note %}}
|
|
To create a robots.txt file without using a template:
|
|
|
|
1. Set `enableRobotsTXT` to `false` in the site configuration.
|
|
2. Create a robots.txt file in the `static` directory.
|
|
|
|
Remember that Hugo copies everything in the [static directory][static] to the root of `publishDir` (typically `public`) when you build your site.
|
|
|
|
[static]: /getting-started/directory-structure/
|
|
{{% /note %}}
|
|
|
|
[site configuration]: /getting-started/configuration/
|