mirror of
https://github.com/gohugoio/hugo.git
synced 2024-11-14 20:37:55 -05:00
ba45da9d03
32356e4e Fix typo in header of shortcode-templates.md c8f1a2d2 Correct code example for index template function bfa6a55d Escape code fencing ff8b2f99 Fix typos in deployment with wercker tutorial 557c36e8 theme: Merge commit '7fbb4bed25001182bfeb91f79db0f0c1936582ee' 7fbb4bed Squashed 'themes/gohugoioTheme/' changes from 7dd8a302..ca53082d ce31cee0 Add "See Also" config 158cee1b Make the tags into keywords 61600be6 Add a note to the related section 49edb5a2 Relase 0.27.1 c9bbc001 releaser: Add release notes to /docs for release of 0.27.1 213c6c3b Add bugs poster 8b4590cd Add KeyCDN integration tutorial 2b277859 Add tutorial videos to several docs pages 950fef1f Update roadmap to link to the correct milestones page 496f5bf6 Rename relnotes d6f9378d Bump Netlify versions to 0.27 087fde7f Update 0.27 release notes 603f94ae docs: Document Related Content 3790f6a3 releaser: Bump versions for release of 0.27 0948868c releaser: Add release notes to /docs for release of 0.27 git-subtree-dir: docs git-subtree-split: 32356e4eabe357ae914f4d1d59e8ae31ce936723
54 lines
No EOL
1.6 KiB
Markdown
54 lines
No EOL
1.6 KiB
Markdown
---
|
|
title: Robots.txt File
|
|
linktitle: Robots.txt
|
|
description: Hugo can generate a customized robots.txt in the same way as any other template.
|
|
date: 2017-02-01
|
|
publishdate: 2017-02-01
|
|
lastmod: 2017-02-01
|
|
categories: [templates]
|
|
keywords: [robots,search engines]
|
|
menu:
|
|
docs:
|
|
parent: "templates"
|
|
weight: 165
|
|
weight: 165
|
|
sections_weight: 165
|
|
draft: false
|
|
aliases: [/extras/robots-txt/]
|
|
toc: false
|
|
---
|
|
|
|
To create your robots.txt as a template, first set the `enableRobotsTXT` value to `true` in your [configuration file][config]. By default, this option generates a robots.txt with the following content, which tells search engines that they are allowed to crawl everything:
|
|
|
|
```
|
|
User-agent: *
|
|
```
|
|
|
|
## Robots.txt Template Lookup Order
|
|
|
|
The [lookup order][lookup] for the `robots.txt` template is as follows:
|
|
|
|
* `/layouts/robots.txt`
|
|
* `/themes/<THEME>/layout/robots.txt`
|
|
|
|
{{% note %}}
|
|
If you do not want Hugo to create a default `robots.txt` or leverage the `robots.txt` template, you can hand code your own and place the file in `static`. Remember that everything in the [static directory](/getting-started/directory-structure/) is copied over as-is when Hugo builds your site.
|
|
{{% /note %}}
|
|
|
|
## Robots.txt Template Example
|
|
|
|
The following is an example `robots.txt` layout:
|
|
|
|
{{< code file="layouts/robots.txt" download="robots.txt" >}}
|
|
User-agent: *
|
|
|
|
{{range .Data.Pages}}
|
|
Disallow: {{.RelPermalink}}
|
|
{{end}}
|
|
{{< /code >}}
|
|
|
|
This template disallows all the pages of the site by creating one `Disallow` entry for each page.
|
|
|
|
[config]: /getting-started/configuration/
|
|
[lookup]: /templates/lookup-order/
|
|
[robots]: http://www.robotstxt.org/ |