hugo/docs/content/extras/robots-txt.md
2016-12-15 11:15:15 +01:00

943 B

lastmod date menu next prev title
2016-01-06 2015-12-08
main
parent
extras
/extras/datafiles /extras/crossreferences Custom robots.txt

Hugo can generated a customized robots.txt in the [same way as any other templates]({{< ref "templates/go-templates.md" >}}).

To enable it, just set enableRobotsTXT option to true in the [configuration file]({{< ref "overview/configuration.md" >}}). By default, it generates a robots.txt, which allows everything, with the following content:

User-agent: *

Hugo will use the template robots.txt according to the following list in descending precedence:

  • /layouts/robots.txt
  • /themes/THEME/layout/robots.txt

An example of a robots.txt layout is:

User-agent: *

{{range .Data.Pages}}
Disallow: {{.RelPermalink}}{{end}}

This template disallows and all the pages of the site creating one Disallow entry for each one.