mirror of
https://github.com/gohugoio/hugo.git
synced 2024-11-29 20:42:07 -05:00
55 lines
1.6 KiB
Markdown
55 lines
1.6 KiB
Markdown
|
---
|
||
|
|
title: Robots.txt File
|
||
|
|
linktitle: Robots.txt
|
||
|
|
description: Hugo can generate a customized robots.txt in the same way as any other template.
|
||
|
|
date: 2017-02-01
|
||
|
|
publishdate: 2017-02-01
|
||
|
|
lastmod: 2017-02-01
|
||
|
|
categories: [templates]
|
||
|
|
keywords: [robots,search engines]
|
||
|
|
menu:
|
||
|
|
docs:
|
||
|
|
parent: "templates"
|
||
|
|
weight: 165
|
||
|
|
weight: 165
|
||
|
|
sections_weight: 165
|
||
|
|
draft: false
|
||
|
|
aliases: [/extras/robots-txt/]
|
||
|
|
toc: false
|
||
|
|
---
|
||
|
|
|
||
|
|
To create your robots.txt as a template, first set the `enableRobotsTXT` value to `true` in your [configuration file][config]. By default, this option generates a robots.txt with the following content, which tells search engines that they are allowed to crawl everything:
|
||
|
|
|
||
|
|
```
|
||
|
|
User-agent: *
|
||
|
|
```
|
||
|
|
|
||
|
|
## Robots.txt Template Lookup Order
|
||
|
|
|
||
|
|
The [lookup order][lookup] for the `robots.txt` template is as follows:
|
||
|
|
|
||
|
|
* `/layouts/robots.txt`
|
||
|
|
* `/themes/<THEME>/layouts/robots.txt`
|
||
|
|
|
||
|
|
{{% note %}}
|
||
|
|
If you do not want Hugo to create a default `robots.txt` or leverage the `robots.txt` template, you can hand code your own and place the file in `static`. Remember that everything in the [static directory](/getting-started/directory-structure/) is copied over as-is when Hugo builds your site.
|
||
|
|
{{% /note %}}
|
||
|
|
|
||
|
|
## Robots.txt Template Example
|
||
|
|
|
||
|
|
The following is an example `robots.txt` layout:
|
||
|
|
|
||
|
|
{{< code file="layouts/robots.txt" download="robots.txt" >}}
|
||
|
|
User-agent: *
|
||
|
|
|
||
|
|
{{range .Pages}}
|
||
|
|
Disallow: {{.RelPermalink}}
|
||
|
|
{{end}}
|
||
|
|
{{< /code >}}
|
||
|
|
|
||
|
|
This template disallows all the pages of the site by creating one `Disallow` entry for each page.
|
||
|
|
|
||
|
|
[config]: /getting-started/configuration/
|
||
|
|
[lookup]: /templates/lookup-order/
|
||
|
|
[robots]: https://www.robotstxt.org/
|