hugo/hugolib/page_test.go

2066 lines
57 KiB
Go
Raw Normal View History

// Copyright 2019 The Hugo Authors. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package hugolib
import (
"context"
"fmt"
2014-01-29 17:50:31 -05:00
"html/template"
"path/filepath"
"strings"
"testing"
"time"
"github.com/bep/clocks"
"github.com/gohugoio/hugo/identity"
Rework external asciidoctor integration This commit solves the relative path problem with asciidoctor tooling. An include will resolve relatively, so you can refer easily to files in the same folder. Also `asciidoctor-diagram` and PlantUML rendering works now, because the created temporary files will be placed in the correct folder. This patch covers just the Ruby version of asciidoctor. The old AsciiDoc CLI EOLs in Jan 2020, so this variant is removed from code. The configuration is completely rewritten and now available in `config.toml` under the key `[markup.asciidocext]`: ```toml [markup.asciidocext] extensions = ["asciidoctor-html5s", "asciidoctor-diagram"] workingFolderCurrent = true trace = true [markup.asciidocext.attributes] my-base-url = "https://example.com/" my-attribute-name = "my value" ``` - backends, safe-modes, and extensions are now whitelisted to the popular (ruby) extensions and valid values. - the default for extensions is to not enable any, because they're all external dependencies so the build would break if the user didn't install them beforehand. - the default backend is html5 because html5s is an external gem dependency. - the default safe-mode is safe, explanations of the modes: https://asciidoctor.org/man/asciidoctor/ - the config is namespaced under asciidocext_config and the parser looks at asciidocext to allow a future native Go asciidoc. - `uglyUrls=true` option and `--source` flag are supported - `--destination` flag is required Follow the updated documentation under `docs/content/en/content-management/formats.md`. This patch would be a breaking change, because you need to correct all your absolute include pathes to relative paths, so using relative paths must be configured explicitly by setting `workingFolderCurrent = true`.
2020-06-25 03:51:33 -04:00
"github.com/gohugoio/hugo/markup/asciidocext"
"github.com/gohugoio/hugo/markup/rst"
"github.com/gohugoio/hugo/tpl"
"github.com/gohugoio/hugo/config"
2022-04-26 13:57:04 -04:00
"github.com/gohugoio/hugo/common/htime"
"github.com/gohugoio/hugo/common/loggers"
"github.com/gohugoio/hugo/resources/page"
"github.com/gohugoio/hugo/resources/resource"
qt "github.com/frankban/quicktest"
"github.com/gohugoio/hugo/deps"
)
const (
homePage = "---\ntitle: Home\n---\nHome Page Content\n"
simplePage = "---\ntitle: Simple\n---\nSimple Page\n"
simplePageRFC3339Date = "---\ntitle: RFC3339 Date\ndate: \"2013-05-17T16:59:30Z\"\n---\nrfc3339 content"
2013-08-07 02:04:49 -04:00
simplePageWithoutSummaryDelimiter = `---
title: SimpleWithoutSummaryDelimiter
---
[Lorem ipsum](https://lipsum.com/) dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Additional text.
Further text.
`
2016-03-24 09:42:03 -04:00
simplePageWithSummaryDelimiter = `---
title: Simple
---
Summary Next Line
<!--more-->
Some more text
`
simplePageWithBlankSummary = `---
title: SimpleWithBlankSummary
---
<!--more-->
Some text.
`
simplePageWithSummaryParameter = `---
title: SimpleWithSummaryParameter
summary: "Page with summary parameter and [a link](http://www.example.com/)"
---
Some text.
Some more text.
`
simplePageWithSummaryDelimiterAndMarkdownThatCrossesBorder = `---
title: Simple
---
The [best static site generator][hugo].[^1]
<!--more-->
[hugo]: http://gohugo.io/
[^1]: Many people say so.
`
2016-03-24 09:42:03 -04:00
simplePageWithShortcodeInSummary = `---
title: Simple
---
Shortcode rewrite, take 2 This commit contains a restructuring and partial rewrite of the shortcode handling. Prior to this commit rendering of the page content was mingled with handling of the shortcodes. This led to several oddities. The new flow is: 1. Shortcodes are extracted from page and replaced with placeholders. 2. Shortcodes are processed and rendered 3. Page is processed 4. The placeholders are replaced with the rendered shortcodes The handling of summaries is also made simpler by this. This commit also introduces some other chenges: 1. distinction between shortcodes that need further processing and those who do not: * `{{< >}}`: Typically raw HTML. Will not be processed. * `{{% %}}`: Will be processed by the page's markup engine (Markdown or (infuture) Asciidoctor) The above also involves a new shortcode-parser, with lexical scanning inspired by Rob Pike's talk called "Lexical Scanning in Go", which should be easier to understand, give better error messages and perform better. 2. If you want to exclude a shortcode from being processed (for documentation etc.), the inner part of the shorcode must be commented out, i.e. `{{%/* movie 47238zzb */%}}`. See the updated shortcode section in the documentation for further examples. The new parser supports nested shortcodes. This isn't new, but has two related design choices worth mentioning: * The shortcodes will be rendered individually, so If both `{{< >}}` and `{{% %}}` are used in the nested hierarchy, one will be passed through the page's markdown processor, the other not. * To avoid potential costly overhead of always looking far ahead for a possible closing tag, this implementation looks at the template itself, and is branded as a container with inner content if it contains a reference to `.Inner` Fixes #565 Fixes #480 Fixes #461 And probably some others.
2014-10-27 16:48:30 -04:00
Summary Next Line. {{<figure src="/not/real" >}}.
More text here.
Some more text
`
2016-03-24 09:42:03 -04:00
simplePageWithSummaryDelimiterSameLine = `---
title: Simple
---
Summary Same Line<!--more-->
Some more text
2015-07-12 05:05:37 -04:00
`
2016-03-24 09:42:03 -04:00
simplePageWithAllCJKRunes = `---
2015-07-12 05:05:37 -04:00
title: Simple
---
你好
도형이
カテゴリー
2015-07-12 05:05:37 -04:00
2013-10-15 09:15:52 -04:00
`
2016-03-24 09:42:03 -04:00
simplePageWithMainEnglishWithCJKRunes = `---
title: Simple
---
In Chinese, means good. In Chinese, means good.
In Chinese, means good. In Chinese, means good.
In Chinese, means good. In Chinese, means good.
In Chinese, means good. In Chinese, means good.
In Chinese, means good. In Chinese, means good.
In Chinese, means good. In Chinese, means good.
In Chinese, means good. In Chinese, means good.
More then 70 words.
`
2016-03-24 09:42:03 -04:00
simplePageWithMainEnglishWithCJKRunesSummary = "In Chinese, 好 means good. In Chinese, 好 means good. " +
"In Chinese, 好 means good. In Chinese, 好 means good. " +
"In Chinese, 好 means good. In Chinese, 好 means good. " +
"In Chinese, 好 means good. In Chinese, 好 means good. " +
"In Chinese, 好 means good. In Chinese, 好 means good. " +
"In Chinese, 好 means good. In Chinese, 好 means good. " +
"In Chinese, 好 means good. In Chinese, 好 means good."
2016-03-24 09:42:03 -04:00
simplePageWithIsCJKLanguageFalse = `---
title: Simple
isCJKLanguage: false
---
In Chinese, 好的啊 means good. In Chinese, 好的呀 means good.
In Chinese, 好的啊 means good. In Chinese, 好的呀 means good.
In Chinese, 好的啊 means good. In Chinese, 好的呀 means good.
In Chinese, 好的啊 means good. In Chinese, 好的呀 means good.
In Chinese, 好的啊 means good. In Chinese, 好的呀 means good.
In Chinese, 好的啊 means good. In Chinese, 好的呀 means good.
In Chinese, 好的啊 means good. In Chinese, 好的呀呀 means good enough.
More then 70 words.
`
2016-03-24 09:42:03 -04:00
simplePageWithIsCJKLanguageFalseSummary = "In Chinese, 好的啊 means good. In Chinese, 好的呀 means good. " +
"In Chinese, 好的啊 means good. In Chinese, 好的呀 means good. " +
"In Chinese, 好的啊 means good. In Chinese, 好的呀 means good. " +
"In Chinese, 好的啊 means good. In Chinese, 好的呀 means good. " +
"In Chinese, 好的啊 means good. In Chinese, 好的呀 means good. " +
"In Chinese, 好的啊 means good. In Chinese, 好的呀 means good. " +
"In Chinese, 好的啊 means good. In Chinese, 好的呀呀 means good enough."
2016-03-24 09:42:03 -04:00
simplePageWithLongContent = `---
2013-10-15 09:15:52 -04:00
title: Simple
---
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis
nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in
culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit
amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore
et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation
ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor
in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui
officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet,
consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et
dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco
laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in
reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia
deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur
adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna
aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi
ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in
voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim
id est laborum. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed
do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim
veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem
ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis
nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in
culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit
amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore
et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation
ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor
in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui
officia deserunt mollit anim id est laborum.`
2016-03-24 09:42:03 -04:00
pageWithToC = `---
title: TOC
---
For some moments the old man did not reply. He stood with bowed head, buried in deep thought. But at last he spoke.
## AA
I have no idea, of course, how long it took me to reach the limit of the plain,
but at last I entered the foothills, following a pretty little canyon upward
toward the mountains. Beside me frolicked a laughing brooklet, hurrying upon
its noisy way down to the silent sea. In its quieter pools I discovered many
small fish, of four-or five-pound weight I should imagine. In appearance,
except as to size and color, they were not unlike the whale of our own seas. As
I watched them playing about I discovered, not only that they suckled their
young, but that at intervals they rose to the surface to breathe as well as to
feed upon certain grasses and a strange, scarlet lichen which grew upon the
rocks just above the water line.
### AAA
I remember I felt an extraordinary persuasion that I was being played with,
that presently, when I was upon the very verge of safety, this mysterious
death--as swift as the passage of light--would leap after me from the pit about
the cylinder and strike me down. ## BB
### BBB
"You're a great Granser," he cried delightedly, "always making believe them little marks mean something."
`
2016-03-24 09:42:03 -04:00
simplePageWithURL = `---
title: Simple
url: simple/url/
---
Simple Page With URL`
2016-03-24 09:42:03 -04:00
simplePageWithSlug = `---
title: Simple
slug: simple-slug
---
Simple Page With Slug`
2016-03-24 09:42:03 -04:00
simplePageWithDate = `---
title: Simple
date: '2013-10-15T06:16:13'
---
Simple Page With Date`
2016-03-24 09:42:03 -04:00
UTF8Page = `---
title: ラーメン
---
UTF8 Page`
2016-03-24 09:42:03 -04:00
UTF8PageWithURL = `---
title: ラーメン
url: ラーメン/url/
---
UTF8 Page With URL`
2016-03-24 09:42:03 -04:00
UTF8PageWithSlug = `---
title: ラーメン
slug: ラーメン-slug
---
UTF8 Page With Slug`
2016-03-24 09:42:03 -04:00
UTF8PageWithDate = `---
title: ラーメン
date: '2013-10-15T06:16:13'
---
UTF8 Page With Date`
)
func checkPageTitle(t *testing.T, page page.Page, title string) {
if page.Title() != title {
t.Fatalf("Page title is: %s. Expected %s", page.Title(), title)
2014-01-29 17:50:31 -05:00
}
}
func checkPageContent(t *testing.T, page page.Page, expected string, msg ...any) {
t.Helper()
a := normalizeContent(expected)
b := normalizeContent(content(page))
if a != b {
t.Fatalf("Page content is:\n%q\nExpected:\n%q (%q)", b, a, msg)
2014-01-29 17:50:31 -05:00
}
}
func normalizeContent(c string) string {
norm := c
norm = strings.Replace(norm, "\n", " ", -1)
norm = strings.Replace(norm, " ", " ", -1)
norm = strings.Replace(norm, " ", " ", -1)
norm = strings.Replace(norm, " ", " ", -1)
norm = strings.Replace(norm, "p> ", "p>", -1)
norm = strings.Replace(norm, "> <", "> <", -1)
return strings.TrimSpace(norm)
}
func checkPageTOC(t *testing.T, page page.Page, toc string) {
t.Helper()
if page.TableOfContents(context.Background()) != template.HTML(toc) {
t.Fatalf("Page TableOfContents is:\n%q.\nExpected %q", page.TableOfContents(context.Background()), toc)
2014-01-29 17:50:31 -05:00
}
}
func checkPageSummary(t *testing.T, page page.Page, summary string, msg ...any) {
a := normalizeContent(string(page.Summary(context.Background())))
b := normalizeContent(summary)
if a != b {
t.Fatalf("Page summary is:\n%q.\nExpected\n%q (%q)", a, b, msg)
2014-01-29 17:50:31 -05:00
}
}
func checkPageType(t *testing.T, page page.Page, pageType string) {
if page.Type() != pageType {
t.Fatalf("Page type is: %s. Expected: %s", page.Type(), pageType)
}
}
func checkPageDate(t *testing.T, page page.Page, time time.Time) {
if page.Date() != time {
t.Fatalf("Page date is: %s. Expected: %s", page.Date(), time)
2014-01-29 17:50:31 -05:00
}
}
func normalizeExpected(ext, str string) string {
str = normalizeContent(str)
switch ext {
default:
return str
case "html":
return strings.Trim(tpl.StripHTML(str), " ")
case "ad":
paragraphs := strings.Split(str, "</p>")
expected := ""
for _, para := range paragraphs {
if para == "" {
continue
}
expected += fmt.Sprintf("<div class=\"paragraph\">\n%s</p></div>\n", para)
}
return expected
case "rst":
if str == "" {
return "<div class=\"document\"></div>"
}
return fmt.Sprintf("<div class=\"document\">\n\n\n%s</div>", str)
}
}
func testAllMarkdownEnginesForPages(t *testing.T,
assertFunc func(t *testing.T, ext string, pages page.Pages), settings map[string]any, pageSources ...string,
) {
engines := []struct {
ext string
shouldExecute func() bool
}{
{"md", func() bool { return true }},
Rework external asciidoctor integration This commit solves the relative path problem with asciidoctor tooling. An include will resolve relatively, so you can refer easily to files in the same folder. Also `asciidoctor-diagram` and PlantUML rendering works now, because the created temporary files will be placed in the correct folder. This patch covers just the Ruby version of asciidoctor. The old AsciiDoc CLI EOLs in Jan 2020, so this variant is removed from code. The configuration is completely rewritten and now available in `config.toml` under the key `[markup.asciidocext]`: ```toml [markup.asciidocext] extensions = ["asciidoctor-html5s", "asciidoctor-diagram"] workingFolderCurrent = true trace = true [markup.asciidocext.attributes] my-base-url = "https://example.com/" my-attribute-name = "my value" ``` - backends, safe-modes, and extensions are now whitelisted to the popular (ruby) extensions and valid values. - the default for extensions is to not enable any, because they're all external dependencies so the build would break if the user didn't install them beforehand. - the default backend is html5 because html5s is an external gem dependency. - the default safe-mode is safe, explanations of the modes: https://asciidoctor.org/man/asciidoctor/ - the config is namespaced under asciidocext_config and the parser looks at asciidocext to allow a future native Go asciidoc. - `uglyUrls=true` option and `--source` flag are supported - `--destination` flag is required Follow the updated documentation under `docs/content/en/content-management/formats.md`. This patch would be a breaking change, because you need to correct all your absolute include pathes to relative paths, so using relative paths must be configured explicitly by setting `workingFolderCurrent = true`.
2020-06-25 03:51:33 -04:00
{"ad", func() bool { return asciidocext.Supports() }},
{"rst", func() bool { return rst.Supports() }},
}
for _, e := range engines {
if !e.shouldExecute() {
continue
}
t.Run(e.ext, func(t *testing.T) {
cfg := config.New()
for k, v := range settings {
cfg.Set(k, v)
}
if s := cfg.GetString("contentDir"); s != "" && s != "content" {
panic("contentDir must be set to 'content' for this test")
}
files := `
-- hugo.toml --
[security]
[security.exec]
allow = ['^python$', '^rst2html.*', '^asciidoctor$']
`
for i, source := range pageSources {
files += fmt.Sprintf("-- content/p%d.%s --\n%s\n", i, e.ext, source)
}
homePath := fmt.Sprintf("_index.%s", e.ext)
files += fmt.Sprintf("-- content/%s --\n%s\n", homePath, homePage)
b := NewIntegrationTestBuilder(
IntegrationTestConfig{
T: t,
TxtarString: files,
NeedsOsFS: true,
BaseCfg: cfg,
},
).Build()
s := b.H.Sites[0]
b.Assert(len(s.RegularPages()), qt.Equals, len(pageSources))
assertFunc(t, e.ext, s.RegularPages())
home := s.Home()
b.Assert(home, qt.Not(qt.IsNil))
b.Assert(home.File().Path(), qt.Equals, homePath)
b.Assert(content(home), qt.Contains, "Home Page Content")
})
}
}
// Issue #1076
func TestPageWithDelimiterForMarkdownThatCrossesBorder(t *testing.T) {
t.Parallel()
cfg, fs := newTestCfg()
c := qt.New(t)
configs, err := loadTestConfigFromProvider(cfg)
c.Assert(err, qt.IsNil)
writeSource(t, fs, filepath.Join("content", "simple.md"), simplePageWithSummaryDelimiterAndMarkdownThatCrossesBorder)
s := buildSingleSite(t, deps.DepsCfg{Fs: fs, Configs: configs}, BuildCfg{SkipRender: true})
c.Assert(len(s.RegularPages()), qt.Equals, 1)
p := s.RegularPages()[0]
if p.Summary(context.Background()) != template.HTML(
"<p>The <a href=\"http://gohugo.io/\">best static site generator</a>.<sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\">1</a></sup></p>") {
t.Fatalf("Got summary:\n%q", p.Summary(context.Background()))
}
cnt := content(p)
if cnt != "<p>The <a href=\"http://gohugo.io/\">best static site generator</a>.<sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\">1</a></sup></p>\n<div class=\"footnotes\" role=\"doc-endnotes\">\n<hr>\n<ol>\n<li id=\"fn:1\">\n<p>Many people say so.&#160;<a href=\"#fnref:1\" class=\"footnote-backref\" role=\"doc-backlink\">&#x21a9;&#xfe0e;</a></p>\n</li>\n</ol>\n</div>" {
t.Fatalf("Got content:\n%q", cnt)
}
}
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
func TestPageDatesTerms(t *testing.T) {
t.Parallel()
files := `
-- hugo.toml --
baseURL = "http://example.com/"
-- content/p1.md --
---
title: p1
date: 2022-01-15
lastMod: 2022-01-16
tags: ["a", "b"]
categories: ["c", "d"]
---
p1
-- content/p2.md --
---
title: p2
date: 2017-01-16
lastMod: 2017-01-17
tags: ["a", "c"]
categories: ["c", "e"]
---
p2
-- layouts/_default/list.html --
{{ .Title }}|Date: {{ .Date.Format "2006-01-02" }}|Lastmod: {{ .Lastmod.Format "2006-01-02" }}|
`
b := Test(t, files)
b.AssertFileContent("public/categories/index.html", "Categories|Date: 2022-01-15|Lastmod: 2022-01-16|")
b.AssertFileContent("public/categories/c/index.html", "C|Date: 2022-01-15|Lastmod: 2022-01-16|")
b.AssertFileContent("public/categories/e/index.html", "E|Date: 2017-01-16|Lastmod: 2017-01-17|")
b.AssertFileContent("public/tags/index.html", "Tags|Date: 2022-01-15|Lastmod: 2022-01-16|")
b.AssertFileContent("public/tags/a/index.html", "A|Date: 2022-01-15|Lastmod: 2022-01-16|")
b.AssertFileContent("public/tags/c/index.html", "C|Date: 2017-01-16|Lastmod: 2017-01-17|")
}
func TestPageDatesAllKinds(t *testing.T) {
t.Parallel()
pageContent := `
---
title: Page
date: 2017-01-15
tags: ["hugo"]
categories: ["cool stuff"]
---
`
b := newTestSitesBuilder(t)
b.WithSimpleConfigFile().WithContent("page.md", pageContent)
b.WithContent("blog/page.md", pageContent)
b.CreateSites().Build(BuildCfg{})
b.Assert(len(b.H.Sites), qt.Equals, 1)
s := b.H.Sites[0]
checkDate := func(t time.Time, msg string) {
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
b.Helper()
Introduce a tree map for all content This commit introduces a new data structure to store pages and their resources. This data structure is backed by radix trees. This simplies tree operations, makes all pages a bundle, and paves the way for #6310. It also solves a set of annoying issues (see list below). Not a motivation behind this, but this commit also makes Hugo in general a little bit faster and more memory effective (see benchmarks). Especially for partial rebuilds on content edits, but also when taxonomies is in use. ``` name old time/op new time/op delta SiteNew/Bundle_with_image/Edit-16 1.32ms ± 8% 1.00ms ± 9% -24.42% (p=0.029 n=4+4) SiteNew/Bundle_with_JSON_file/Edit-16 1.28ms ± 0% 0.94ms ± 0% -26.26% (p=0.029 n=4+4) SiteNew/Tags_and_categories/Edit-16 33.9ms ± 2% 21.8ms ± 1% -35.67% (p=0.029 n=4+4) SiteNew/Canonify_URLs/Edit-16 40.6ms ± 1% 37.7ms ± 3% -7.20% (p=0.029 n=4+4) SiteNew/Deep_content_tree/Edit-16 56.7ms ± 0% 51.7ms ± 1% -8.82% (p=0.029 n=4+4) SiteNew/Many_HTML_templates/Edit-16 19.9ms ± 2% 18.3ms ± 3% -7.64% (p=0.029 n=4+4) SiteNew/Page_collections/Edit-16 37.9ms ± 4% 34.0ms ± 2% -10.28% (p=0.029 n=4+4) SiteNew/Bundle_with_image-16 10.7ms ± 0% 10.6ms ± 0% -1.15% (p=0.029 n=4+4) SiteNew/Bundle_with_JSON_file-16 10.8ms ± 0% 10.7ms ± 0% -1.05% (p=0.029 n=4+4) SiteNew/Tags_and_categories-16 43.2ms ± 1% 39.6ms ± 1% -8.35% (p=0.029 n=4+4) SiteNew/Canonify_URLs-16 47.6ms ± 1% 47.3ms ± 0% ~ (p=0.057 n=4+4) SiteNew/Deep_content_tree-16 73.0ms ± 1% 74.2ms ± 1% ~ (p=0.114 n=4+4) SiteNew/Many_HTML_templates-16 37.9ms ± 0% 38.1ms ± 1% ~ (p=0.114 n=4+4) SiteNew/Page_collections-16 53.6ms ± 1% 54.7ms ± 1% +2.09% (p=0.029 n=4+4) name old alloc/op new alloc/op delta SiteNew/Bundle_with_image/Edit-16 486kB ± 0% 430kB ± 0% -11.47% (p=0.029 n=4+4) SiteNew/Bundle_with_JSON_file/Edit-16 265kB ± 0% 209kB ± 0% -21.06% (p=0.029 n=4+4) SiteNew/Tags_and_categories/Edit-16 13.6MB ± 0% 8.8MB ± 0% -34.93% (p=0.029 n=4+4) SiteNew/Canonify_URLs/Edit-16 66.5MB ± 0% 63.9MB ± 0% -3.95% (p=0.029 n=4+4) SiteNew/Deep_content_tree/Edit-16 28.8MB ± 0% 25.8MB ± 0% -10.55% (p=0.029 n=4+4) SiteNew/Many_HTML_templates/Edit-16 6.16MB ± 0% 5.56MB ± 0% -9.86% (p=0.029 n=4+4) SiteNew/Page_collections/Edit-16 16.9MB ± 0% 16.0MB ± 0% -5.19% (p=0.029 n=4+4) SiteNew/Bundle_with_image-16 2.28MB ± 0% 2.29MB ± 0% +0.35% (p=0.029 n=4+4) SiteNew/Bundle_with_JSON_file-16 2.07MB ± 0% 2.07MB ± 0% ~ (p=0.114 n=4+4) SiteNew/Tags_and_categories-16 14.3MB ± 0% 13.2MB ± 0% -7.30% (p=0.029 n=4+4) SiteNew/Canonify_URLs-16 69.1MB ± 0% 69.0MB ± 0% ~ (p=0.343 n=4+4) SiteNew/Deep_content_tree-16 31.3MB ± 0% 31.8MB ± 0% +1.49% (p=0.029 n=4+4) SiteNew/Many_HTML_templates-16 10.8MB ± 0% 10.9MB ± 0% +1.11% (p=0.029 n=4+4) SiteNew/Page_collections-16 21.4MB ± 0% 21.6MB ± 0% +1.15% (p=0.029 n=4+4) name old allocs/op new allocs/op delta SiteNew/Bundle_with_image/Edit-16 4.74k ± 0% 3.86k ± 0% -18.57% (p=0.029 n=4+4) SiteNew/Bundle_with_JSON_file/Edit-16 4.73k ± 0% 3.85k ± 0% -18.58% (p=0.029 n=4+4) SiteNew/Tags_and_categories/Edit-16 301k ± 0% 198k ± 0% -34.14% (p=0.029 n=4+4) SiteNew/Canonify_URLs/Edit-16 389k ± 0% 373k ± 0% -4.07% (p=0.029 n=4+4) SiteNew/Deep_content_tree/Edit-16 338k ± 0% 262k ± 0% -22.63% (p=0.029 n=4+4) SiteNew/Many_HTML_templates/Edit-16 102k ± 0% 88k ± 0% -13.81% (p=0.029 n=4+4) SiteNew/Page_collections/Edit-16 176k ± 0% 152k ± 0% -13.32% (p=0.029 n=4+4) SiteNew/Bundle_with_image-16 26.8k ± 0% 26.8k ± 0% +0.05% (p=0.029 n=4+4) SiteNew/Bundle_with_JSON_file-16 26.8k ± 0% 26.8k ± 0% +0.05% (p=0.029 n=4+4) SiteNew/Tags_and_categories-16 273k ± 0% 245k ± 0% -10.36% (p=0.029 n=4+4) SiteNew/Canonify_URLs-16 396k ± 0% 398k ± 0% +0.39% (p=0.029 n=4+4) SiteNew/Deep_content_tree-16 317k ± 0% 325k ± 0% +2.53% (p=0.029 n=4+4) SiteNew/Many_HTML_templates-16 146k ± 0% 147k ± 0% +0.98% (p=0.029 n=4+4) SiteNew/Page_collections-16 210k ± 0% 215k ± 0% +2.44% (p=0.029 n=4+4) ``` Fixes #6312 Fixes #6087 Fixes #6738 Fixes #6412 Fixes #6743 Fixes #6875 Fixes #6034 Fixes #6902 Fixes #6173 Fixes #6590
2019-09-10 05:26:34 -04:00
b.Assert(t.Year(), qt.Equals, 2017, qt.Commentf(msg))
}
checkDated := func(d resource.Dated, msg string) {
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
b.Helper()
checkDate(d.Date(), "date: "+msg)
checkDate(d.Lastmod(), "lastmod: "+msg)
}
for _, p := range s.Pages() {
checkDated(p, p.Kind())
}
checkDate(s.LastChange(), "site")
}
func TestPageDatesSections(t *testing.T) {
t.Parallel()
b := newTestSitesBuilder(t)
b.WithSimpleConfigFile().WithContent("no-index/page.md", `
---
title: Page
date: 2017-01-15
---
`, "with-index-no-date/_index.md", `---
title: No Date
---
`,
// https://github.com/gohugoio/hugo/issues/5854
"with-index-date/_index.md", `---
title: Date
date: 2018-01-15
---
`, "with-index-date/p1.md", `---
title: Date
date: 2018-01-15
---
`, "with-index-date/p1.md", `---
title: Date
date: 2018-01-15
---
`)
for i := 1; i <= 20; i++ {
b.WithContent(fmt.Sprintf("main-section/p%d.md", i), `---
title: Date
date: 2012-01-12
---
`)
}
b.CreateSites().Build(BuildCfg{})
b.Assert(len(b.H.Sites), qt.Equals, 1)
s := b.H.Sites[0]
checkDate := func(p page.Page, year int) {
b.Assert(p.Date().Year(), qt.Equals, year)
b.Assert(p.Lastmod().Year(), qt.Equals, year)
}
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
checkDate(s.getPageOldVersion("/"), 2018)
checkDate(s.getPageOldVersion("/no-index"), 2017)
b.Assert(s.getPageOldVersion("/with-index-no-date").Date().IsZero(), qt.Equals, true)
checkDate(s.getPageOldVersion("/with-index-date"), 2018)
b.Assert(s.Site().Lastmod().Year(), qt.Equals, 2018)
}
func TestCreateNewPage(t *testing.T) {
t.Parallel()
c := qt.New(t)
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
p := pages[0]
// issue #2290: Path is relative to the content dir and will continue to be so.
c.Assert(p.File().Path(), qt.Equals, fmt.Sprintf("p0.%s", ext))
c.Assert(p.IsHome(), qt.Equals, false)
checkPageTitle(t, p, "Simple")
checkPageContent(t, p, normalizeExpected(ext, "<p>Simple Page</p>\n"))
checkPageSummary(t, p, "Simple Page")
checkPageType(t, p, "page")
2014-01-29 17:50:31 -05:00
}
settings := map[string]any{}
testAllMarkdownEnginesForPages(t, assertFunc, settings, simplePage)
}
func TestPageSummary(t *testing.T) {
t.Parallel()
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
p := pages[0]
checkPageTitle(t, p, "SimpleWithoutSummaryDelimiter")
2019-07-31 08:36:36 -04:00
// Source is not Asciidoctor- or RST-compatible so don't test them
if ext != "ad" && ext != "rst" {
checkPageContent(t, p, normalizeExpected(ext, "<p><a href=\"https://lipsum.com/\">Lorem ipsum</a> dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>\n\n<p>Additional text.</p>\n\n<p>Further text.</p>\n"), ext)
checkPageSummary(t, p, normalizeExpected(ext, "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Additional text."), ext)
}
checkPageType(t, p, "page")
}
testAllMarkdownEnginesForPages(t, assertFunc, nil, simplePageWithoutSummaryDelimiter)
}
func TestPageWithDelimiter(t *testing.T) {
t.Parallel()
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
p := pages[0]
checkPageTitle(t, p, "Simple")
checkPageContent(t, p, normalizeExpected(ext, "<p>Summary Next Line</p>\n\n<p>Some more text</p>\n"), ext)
checkPageSummary(t, p, normalizeExpected(ext, "<p>Summary Next Line</p>"), ext)
checkPageType(t, p, "page")
}
testAllMarkdownEnginesForPages(t, assertFunc, nil, simplePageWithSummaryDelimiter)
}
func TestPageWithBlankSummary(t *testing.T) {
t.Parallel()
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
p := pages[0]
checkPageTitle(t, p, "SimpleWithBlankSummary")
checkPageContent(t, p, normalizeExpected(ext, "<p>Some text.</p>\n"), ext)
checkPageSummary(t, p, normalizeExpected(ext, ""), ext)
checkPageType(t, p, "page")
}
testAllMarkdownEnginesForPages(t, assertFunc, nil, simplePageWithBlankSummary)
}
func TestPageWithSummaryParameter(t *testing.T) {
t.Parallel()
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
p := pages[0]
checkPageTitle(t, p, "SimpleWithSummaryParameter")
checkPageContent(t, p, normalizeExpected(ext, "<p>Some text.</p>\n\n<p>Some more text.</p>\n"), ext)
2019-07-31 08:36:36 -04:00
// Summary is not Asciidoctor- or RST-compatible so don't test them
if ext != "ad" && ext != "rst" {
checkPageSummary(t, p, normalizeExpected(ext, "Page with summary parameter and <a href=\"http://www.example.com/\">a link</a>"), ext)
}
checkPageType(t, p, "page")
}
testAllMarkdownEnginesForPages(t, assertFunc, nil, simplePageWithSummaryParameter)
}
// Issue #3854
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
// Also see https://github.com/gohugoio/hugo/issues/3977
func TestPageWithDateFields(t *testing.T) {
c := qt.New(t)
pageWithDate := `---
title: P%d
weight: %d
%s: 2017-10-13
---
Simple Page With Some Date`
hasDate := func(p page.Page) bool {
return p.Date().Year() == 2017
}
datePage := func(field string, weight int) string {
return fmt.Sprintf(pageWithDate, weight, weight, field)
}
t.Parallel()
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
c.Assert(len(pages) > 0, qt.Equals, true)
for _, p := range pages {
c.Assert(hasDate(p), qt.Equals, true)
}
}
fields := []string{"date", "publishdate", "pubdate", "published"}
pageContents := make([]string, len(fields))
for i, field := range fields {
pageContents[i] = datePage(field, i+1)
}
testAllMarkdownEnginesForPages(t, assertFunc, nil, pageContents...)
}
func TestPageRawContent(t *testing.T) {
files := `
-- hugo.toml --
-- content/basic.md --
---
title: "basic"
---
**basic**
-- content/empty.md --
---
title: "empty"
---
-- layouts/_default/single.html --
|{{ .RawContent }}|
`
b := Test(t, files)
b.AssertFileContent("public/basic/index.html", "|**basic**|")
b.AssertFileContent("public/empty/index.html", "! title")
}
func TestPageWithShortCodeInSummary(t *testing.T) {
t.Parallel()
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
p := pages[0]
checkPageTitle(t, p, "Simple")
checkPageContent(t, p, normalizeExpected(ext, "<p>Summary Next Line. <figure><img src=\"/not/real\"> </figure> . More text here.</p><p>Some more text</p>"))
checkPageSummary(t, p, "Summary Next Line. . More text here. Some more text")
checkPageType(t, p, "page")
2014-01-29 17:50:31 -05:00
}
Shortcode rewrite, take 2 This commit contains a restructuring and partial rewrite of the shortcode handling. Prior to this commit rendering of the page content was mingled with handling of the shortcodes. This led to several oddities. The new flow is: 1. Shortcodes are extracted from page and replaced with placeholders. 2. Shortcodes are processed and rendered 3. Page is processed 4. The placeholders are replaced with the rendered shortcodes The handling of summaries is also made simpler by this. This commit also introduces some other chenges: 1. distinction between shortcodes that need further processing and those who do not: * `{{< >}}`: Typically raw HTML. Will not be processed. * `{{% %}}`: Will be processed by the page's markup engine (Markdown or (infuture) Asciidoctor) The above also involves a new shortcode-parser, with lexical scanning inspired by Rob Pike's talk called "Lexical Scanning in Go", which should be easier to understand, give better error messages and perform better. 2. If you want to exclude a shortcode from being processed (for documentation etc.), the inner part of the shorcode must be commented out, i.e. `{{%/* movie 47238zzb */%}}`. See the updated shortcode section in the documentation for further examples. The new parser supports nested shortcodes. This isn't new, but has two related design choices worth mentioning: * The shortcodes will be rendered individually, so If both `{{< >}}` and `{{% %}}` are used in the nested hierarchy, one will be passed through the page's markdown processor, the other not. * To avoid potential costly overhead of always looking far ahead for a possible closing tag, this implementation looks at the template itself, and is branded as a container with inner content if it contains a reference to `.Inner` Fixes #565 Fixes #480 Fixes #461 And probably some others.
2014-10-27 16:48:30 -04:00
testAllMarkdownEnginesForPages(t, assertFunc, nil, simplePageWithShortcodeInSummary)
}
func TestTableOfContents(t *testing.T) {
c := qt.New(t)
cfg, fs := newTestCfg()
configs, err := loadTestConfigFromProvider(cfg)
c.Assert(err, qt.IsNil)
writeSource(t, fs, filepath.Join("content", "tocpage.md"), pageWithToC)
s := buildSingleSite(t, deps.DepsCfg{Fs: fs, Configs: configs}, BuildCfg{SkipRender: true})
c.Assert(len(s.RegularPages()), qt.Equals, 1)
p := s.RegularPages()[0]
checkPageContent(t, p, "<p>For some moments the old man did not reply. He stood with bowed head, buried in deep thought. But at last he spoke.</p><h2 id=\"aa\">AA</h2> <p>I have no idea, of course, how long it took me to reach the limit of the plain, but at last I entered the foothills, following a pretty little canyon upward toward the mountains. Beside me frolicked a laughing brooklet, hurrying upon its noisy way down to the silent sea. In its quieter pools I discovered many small fish, of four-or five-pound weight I should imagine. In appearance, except as to size and color, they were not unlike the whale of our own seas. As I watched them playing about I discovered, not only that they suckled their young, but that at intervals they rose to the surface to breathe as well as to feed upon certain grasses and a strange, scarlet lichen which grew upon the rocks just above the water line.</p><h3 id=\"aaa\">AAA</h3> <p>I remember I felt an extraordinary persuasion that I was being played with, that presently, when I was upon the very verge of safety, this mysterious death&ndash;as swift as the passage of light&ndash;would leap after me from the pit about the cylinder and strike me down. ## BB</p><h3 id=\"bbb\">BBB</h3> <p>&ldquo;You&rsquo;re a great Granser,&rdquo; he cried delightedly, &ldquo;always making believe them little marks mean something.&rdquo;</p>")
checkPageTOC(t, p, "<nav id=\"TableOfContents\">\n <ul>\n <li><a href=\"#aa\">AA</a>\n <ul>\n <li><a href=\"#aaa\">AAA</a></li>\n <li><a href=\"#bbb\">BBB</a></li>\n </ul>\n </li>\n </ul>\n</nav>")
}
func TestPageWithMoreTag(t *testing.T) {
t.Parallel()
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
p := pages[0]
checkPageTitle(t, p, "Simple")
checkPageContent(t, p, normalizeExpected(ext, "<p>Summary Same Line</p>\n\n<p>Some more text</p>\n"))
checkPageSummary(t, p, normalizeExpected(ext, "<p>Summary Same Line</p>"))
checkPageType(t, p, "page")
2014-01-29 17:50:31 -05:00
}
testAllMarkdownEnginesForPages(t, assertFunc, nil, simplePageWithSummaryDelimiterSameLine)
}
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
func TestSummaryInFrontMatter(t *testing.T) {
t.Parallel()
Test(t, `
-- hugo.toml --
-- content/simple.md --
---
title: Simple
summary: "Front **matter** summary"
---
Simple Page
-- layouts/_default/single.html --
Summary: {{ .Summary }}|Truncated: {{ .Truncated }}|
`).AssertFileContent("public/simple/index.html", "Summary: Front <strong>matter</strong> summary|", "Truncated: false")
}
func TestSummaryManualSplit(t *testing.T) {
t.Parallel()
Test(t, `
-- hugo.toml --
-- content/simple.md --
---
title: Simple
---
This is **summary**.
<!--more-->
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
This is **content**.
-- layouts/_default/single.html --
Summary: {{ .Summary }}|Truncated: {{ .Truncated }}|
Content: {{ .Content }}|
`).AssertFileContent("public/simple/index.html",
"Summary: <p>This is <strong>summary</strong>.</p>|",
"Truncated: true|",
"Content: <p>This is <strong>summary</strong>.</p>\n<p>This is <strong>content</strong>.</p>|",
)
}
func TestSummaryManualSplitHTML(t *testing.T) {
t.Parallel()
Test(t, `
-- hugo.toml --
-- content/simple.html --
---
title: Simple
---
<div>
This is <b>summary</b>.
</div>
<!--more-->
<div>
This is <b>content</b>.
</div>
-- layouts/_default/single.html --
Summary: {{ .Summary }}|Truncated: {{ .Truncated }}|
Content: {{ .Content }}|
`).AssertFileContent("public/simple/index.html", "Summary: <div>\nThis is <b>summary</b>.\n</div>\n|Truncated: true|\nContent: \n\n<div>\nThis is <b>content</b>.\n</div>|")
}
func TestSummaryAuto(t *testing.T) {
t.Parallel()
Test(t, `
-- hugo.toml --
summaryLength = 10
-- content/simple.md --
---
title: Simple
---
This is **summary**.
This is **more summary**.
This is *even more summary**.
This is **more summary**.
This is **content**.
-- layouts/_default/single.html --
Summary: {{ .Summary }}|Truncated: {{ .Truncated }}|
Content: {{ .Content }}|
`).AssertFileContent("public/simple/index.html",
"Summary: This is summary. This is more summary. This is even more summary*.|",
"Truncated: true|",
"Content: <p>This is <strong>summary</strong>.")
}
// #2973
func TestSummaryWithHTMLTagsOnNextLine(t *testing.T) {
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
c := qt.New(t)
p := pages[0]
s := string(p.Summary(context.Background()))
c.Assert(s, qt.Contains, "Happy new year everyone!")
c.Assert(s, qt.Not(qt.Contains), "User interface")
}
testAllMarkdownEnginesForPages(t, assertFunc, nil, `---
title: Simple
---
Happy new year everyone!
Here is the last report for commits in the year 2016. It covers hrev50718-hrev50829.
<!--more-->
<h3>User interface</h3>
`)
}
// Issue 9383
func TestRenderStringForRegularPageTranslations(t *testing.T) {
c := qt.New(t)
b := newTestSitesBuilder(t)
b.WithLogger(loggers.NewDefault())
b.WithConfigFile("toml",
`baseurl = "https://example.org/"
title = "My Site"
defaultContentLanguage = "ru"
defaultContentLanguageInSubdir = true
[languages.ru]
contentDir = 'content/ru'
weight = 1
[languages.en]
weight = 2
contentDir = 'content/en'
[outputs]
home = ["HTML", "JSON"]`)
b.WithTemplates("index.html", `
{{- range .Site.Home.Translations -}}
<p>{{- .RenderString "foo" -}}</p>
{{- end -}}
{{- range .Site.Home.AllTranslations -}}
<p>{{- .RenderString "bar" -}}</p>
{{- end -}}
`, "_default/single.html",
`{{ .Content }}`,
"index.json",
`{"Title": "My Site"}`,
)
b.WithContent(
"ru/a.md",
"",
"en/a.md",
"",
)
err := b.BuildE(BuildCfg{})
c.Assert(err, qt.Equals, nil)
b.AssertFileContent("public/ru/index.html", `
<p>foo</p>
<p>foo</p>
<p>bar</p>
<p>bar</p>
`)
b.AssertFileContent("public/en/index.html", `
<p>foo</p>
<p>foo</p>
<p>bar</p>
<p>bar</p>
`)
}
// Issue 8919
func TestContentProviderWithCustomOutputFormat(t *testing.T) {
b := newTestSitesBuilder(t)
b.WithLogger(loggers.NewDefault())
b.WithConfigFile("toml", `baseURL = 'http://example.org/'
title = 'My New Hugo Site'
timeout = 600000 # ten minutes in case we want to pause and debug
defaultContentLanguage = "en"
[languages]
[languages.en]
title = "Repro"
languageName = "English"
contentDir = "content/en"
[languages.zh_CN]
title = "Repro"
languageName = "简体中文"
contentDir = "content/zh_CN"
[outputFormats]
[outputFormats.metadata]
baseName = "metadata"
mediaType = "text/html"
isPlainText = true
notAlternative = true
[outputs]
home = ["HTML", "metadata"]`)
b.WithTemplates("home.metadata.html", `<h2>Translations metadata</h2>
<ul>
{{ $p := .Page }}
{{ range $p.Translations}}
<li>Title: {{ .Title }}, {{ .Summary }}</li>
<li>Content: {{ .Content }}</li>
<li>Plain: {{ .Plain }}</li>
<li>PlainWords: {{ .PlainWords }}</li>
<li>Summary: {{ .Summary }}</li>
<li>Truncated: {{ .Truncated }}</li>
<li>FuzzyWordCount: {{ .FuzzyWordCount }}</li>
<li>ReadingTime: {{ .ReadingTime }}</li>
<li>Len: {{ .Len }}</li>
{{ end }}
</ul>`)
b.WithTemplates("_default/baseof.html", `<html>
<body>
{{ block "main" . }}{{ end }}
</body>
</html>`)
b.WithTemplates("_default/home.html", `{{ define "main" }}
<h2>Translations</h2>
<ul>
{{ $p := .Page }}
{{ range $p.Translations}}
<li>Title: {{ .Title }}, {{ .Summary }}</li>
<li>Content: {{ .Content }}</li>
<li>Plain: {{ .Plain }}</li>
<li>PlainWords: {{ .PlainWords }}</li>
<li>Summary: {{ .Summary }}</li>
<li>Truncated: {{ .Truncated }}</li>
<li>FuzzyWordCount: {{ .FuzzyWordCount }}</li>
<li>ReadingTime: {{ .ReadingTime }}</li>
<li>Len: {{ .Len }}</li>
{{ end }}
</ul>
{{ end }}`)
b.WithContent("en/_index.md", `---
title: Title (en)
summary: Summary (en)
---
Here is some content.
`)
b.WithContent("zh_CN/_index.md", `---
title: Title (zh)
summary: Summary (zh)
---
这是一些内容
`)
b.Build(BuildCfg{})
b.AssertFileContent("public/index.html", `<html>
<body>
<h2>Translations</h2>
<ul>
<li>Title: Title (zh), Summary (zh)</li>
<li>Content: <p>这是一些内容</p>
</li>
<li>Plain: 这是一些内容
</li>
<li>PlainWords: [这是一些内容]</li>
<li>Summary: Summary (zh)</li>
<li>Truncated: false</li>
<li>FuzzyWordCount: 100</li>
<li>ReadingTime: 1</li>
<li>Len: 26</li>
</ul>
</body>
</html>`)
b.AssertFileContent("public/metadata.html", `<h2>Translations metadata</h2>
<ul>
<li>Title: Title (zh), Summary (zh)</li>
<li>Content: <p>这是一些内容</p>
</li>
<li>Plain: 这是一些内容
</li>
<li>PlainWords: [这是一些内容]</li>
<li>Summary: Summary (zh)</li>
<li>Truncated: false</li>
<li>FuzzyWordCount: 100</li>
<li>ReadingTime: 1</li>
<li>Len: 26</li>
</ul>`)
b.AssertFileContent("public/zh_cn/index.html", `<html>
<body>
<h2>Translations</h2>
<ul>
<li>Title: Title (en), Summary (en)</li>
<li>Content: <p>Here is some content.</p>
</li>
<li>Plain: Here is some content.
</li>
<li>PlainWords: [Here is some content.]</li>
<li>Summary: Summary (en)</li>
<li>Truncated: false</li>
<li>FuzzyWordCount: 100</li>
<li>ReadingTime: 1</li>
<li>Len: 29</li>
</ul>
</body>
</html>`)
b.AssertFileContent("public/zh_cn/metadata.html", `<h2>Translations metadata</h2>
<ul>
<li>Title: Title (en), Summary (en)</li>
<li>Content: <p>Here is some content.</p>
</li>
<li>Plain: Here is some content.
</li>
<li>PlainWords: [Here is some content.]</li>
<li>Summary: Summary (en)</li>
<li>Truncated: false</li>
<li>FuzzyWordCount: 100</li>
<li>ReadingTime: 1</li>
<li>Len: 29</li>
</ul>`)
}
func TestPageWithDate(t *testing.T) {
t.Parallel()
c := qt.New(t)
cfg, fs := newTestCfg()
configs, err := loadTestConfigFromProvider(cfg)
c.Assert(err, qt.IsNil)
writeSource(t, fs, filepath.Join("content", "simple.md"), simplePageRFC3339Date)
s := buildSingleSite(t, deps.DepsCfg{Fs: fs, Configs: configs}, BuildCfg{SkipRender: true})
c.Assert(len(s.RegularPages()), qt.Equals, 1)
p := s.RegularPages()[0]
d, _ := time.Parse(time.RFC3339, "2013-05-17T16:59:30Z")
2014-01-29 17:50:31 -05:00
checkPageDate(t, p, d)
}
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
func TestPageWithFrontMatterConfig(t *testing.T) {
for _, dateHandler := range []string{":filename", ":fileModTime"} {
dateHandler := dateHandler
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
t.Run(fmt.Sprintf("dateHandler=%q", dateHandler), func(t *testing.T) {
t.Parallel()
c := qt.New(t)
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
cfg, fs := newTestCfg()
2018-01-24 09:21:55 -05:00
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
pageTemplate := `
2018-01-24 09:21:55 -05:00
---
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
title: Page
weight: %d
lastMod: 2018-02-28
%s
2018-01-24 09:21:55 -05:00
---
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
Content
`
2018-01-24 09:21:55 -05:00
cfg.Set("frontmatter", map[string]any{
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
"date": []string{dateHandler, "date"},
})
configs, err := loadTestConfigFromProvider(cfg)
c.Assert(err, qt.IsNil)
2018-01-24 09:21:55 -05:00
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
c1 := filepath.Join("content", "section", "2012-02-21-noslug.md")
c2 := filepath.Join("content", "section", "2012-02-22-slug.md")
2018-01-24 09:21:55 -05:00
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
writeSource(t, fs, c1, fmt.Sprintf(pageTemplate, 1, ""))
writeSource(t, fs, c2, fmt.Sprintf(pageTemplate, 2, "slug: aslug"))
2018-01-24 09:21:55 -05:00
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
c1fi, err := fs.Source.Stat(c1)
c.Assert(err, qt.IsNil)
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
c2fi, err := fs.Source.Stat(c2)
c.Assert(err, qt.IsNil)
b := newTestSitesBuilderFromDepsCfg(t, deps.DepsCfg{Fs: fs, Configs: configs}).WithNothingAdded()
b.Build(BuildCfg{SkipRender: true})
2018-01-24 09:21:55 -05:00
s := b.H.Sites[0]
c.Assert(len(s.RegularPages()), qt.Equals, 2)
2018-01-24 09:21:55 -05:00
noSlug := s.RegularPages()[0]
slug := s.RegularPages()[1]
2018-01-24 09:21:55 -05:00
c.Assert(noSlug.Lastmod().Day(), qt.Equals, 28)
2018-01-24 09:21:55 -05:00
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
switch strings.ToLower(dateHandler) {
case ":filename":
c.Assert(noSlug.Date().IsZero(), qt.Equals, false)
c.Assert(slug.Date().IsZero(), qt.Equals, false)
c.Assert(noSlug.Date().Year(), qt.Equals, 2012)
c.Assert(slug.Date().Year(), qt.Equals, 2012)
c.Assert(noSlug.Slug(), qt.Equals, "noslug")
c.Assert(slug.Slug(), qt.Equals, "aslug")
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
case ":filemodtime":
c.Assert(noSlug.Date().Year(), qt.Equals, c1fi.ModTime().Year())
c.Assert(slug.Date().Year(), qt.Equals, c2fi.ModTime().Year())
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
fallthrough
default:
c.Assert(noSlug.Slug(), qt.Equals, "")
c.Assert(slug.Slug(), qt.Equals, "aslug")
2018-01-24 09:21:55 -05:00
hugolib: Extract date and slug from filename This commit makes it possible to extract the date from the content filename. Also, the filenames in these cases will make for very poor permalinks, so we will also use the remaining part as the page `slug` if that value is not set in front matter. This should make it easier to move content from Jekyll to Hugo. To enable, put this in your `config.toml`: ```toml [frontmatter] date = [":filename", ":default"] ``` This commit is also a spring cleaning of how the different dates are configured in Hugo. Hugo will check for dates following the configuration from left to right, starting with `:filename` etc. So, if you want to use the `file modification time`, this can be a good configuration: ```toml [frontmatter] date = [ "date",":fileModTime", ":default"] lastmod = ["lastmod" ,":fileModTime", ":default"] ``` The current `:default` values for the different dates are ```toml [frontmatter] date = ["date","publishDate", "lastmod"] lastmod = ["lastmod", "date","publishDate"] publishDate = ["publishDate", "date"] expiryDate = ["expiryDate"] ``` The above will now be the same as: ```toml [frontmatter] date = [":default"] lastmod = [":default"] publishDate = [":default"] expiryDate = [":default"] ``` Note: * We have some built-in aliases to the above: lastmod => modified, publishDate => pubdate, published and expiryDate => unpublishdate. * If you want a new configuration for, say, `date`, you can provide only that line, and the rest will be preserved. * All the keywords to the right that does not start with a ":" maps to front matter parameters, and can be any date param (e.g. `myCustomDateParam`). * The keywords to the left are the **4 predefined dates in Hugo**, i.e. they are constant values. * The current "special date handlers" are `:fileModTime` and `:filename`. We will soon add `:git` to that list. Fixes #285 Closes #3310 Closes #3762 Closes #4340
2018-03-11 06:32:55 -04:00
}
})
2018-01-24 09:21:55 -05:00
}
}
func TestWordCountWithAllCJKRunesWithoutHasCJKLanguage(t *testing.T) {
t.Parallel()
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
p := pages[0]
if p.WordCount(context.Background()) != 8 {
t.Fatalf("[%s] incorrect word count. expected %v, got %v", ext, 8, p.WordCount(context.Background()))
}
}
testAllMarkdownEnginesForPages(t, assertFunc, nil, simplePageWithAllCJKRunes)
}
func TestWordCountWithAllCJKRunesHasCJKLanguage(t *testing.T) {
t.Parallel()
settings := map[string]any{"hasCJKLanguage": true}
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
p := pages[0]
if p.WordCount(context.Background()) != 15 {
t.Fatalf("[%s] incorrect word count, expected %v, got %v", ext, 15, p.WordCount(context.Background()))
}
}
testAllMarkdownEnginesForPages(t, assertFunc, settings, simplePageWithAllCJKRunes)
}
func TestWordCountWithMainEnglishWithCJKRunes(t *testing.T) {
t.Parallel()
settings := map[string]any{"hasCJKLanguage": true}
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
p := pages[0]
if p.WordCount(context.Background()) != 74 {
t.Fatalf("[%s] incorrect word count, expected %v, got %v", ext, 74, p.WordCount(context.Background()))
}
2015-07-12 05:05:37 -04:00
if p.Summary(context.Background()) != simplePageWithMainEnglishWithCJKRunesSummary {
t.Fatalf("[%s] incorrect Summary for content '%s'. expected\n%v, got\n%v", ext, p.Plain(context.Background()),
simplePageWithMainEnglishWithCJKRunesSummary, p.Summary(context.Background()))
}
}
testAllMarkdownEnginesForPages(t, assertFunc, settings, simplePageWithMainEnglishWithCJKRunes)
}
func TestWordCountWithIsCJKLanguageFalse(t *testing.T) {
t.Parallel()
settings := map[string]any{
"hasCJKLanguage": true,
}
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
p := pages[0]
if p.WordCount(context.Background()) != 75 {
t.Fatalf("[%s] incorrect word count for content '%s'. expected %v, got %v", ext, p.Plain(context.Background()), 74, p.WordCount(context.Background()))
}
if p.Summary(context.Background()) != simplePageWithIsCJKLanguageFalseSummary {
t.Fatalf("[%s] incorrect Summary for content '%s'. expected %v, got %v", ext, p.Plain(context.Background()),
simplePageWithIsCJKLanguageFalseSummary, p.Summary(context.Background()))
}
2015-07-12 05:05:37 -04:00
}
testAllMarkdownEnginesForPages(t, assertFunc, settings, simplePageWithIsCJKLanguageFalse)
2015-07-12 05:05:37 -04:00
}
2013-10-15 09:15:52 -04:00
func TestWordCount(t *testing.T) {
t.Parallel()
assertFunc := func(t *testing.T, ext string, pages page.Pages) {
p := pages[0]
if p.WordCount(context.Background()) != 483 {
t.Fatalf("[%s] incorrect word count. expected %v, got %v", ext, 483, p.WordCount(context.Background()))
}
2014-01-29 17:50:31 -05:00
if p.FuzzyWordCount(context.Background()) != 500 {
t.Fatalf("[%s] incorrect word count. expected %v, got %v", ext, 500, p.FuzzyWordCount(context.Background()))
}
if p.ReadingTime(context.Background()) != 3 {
t.Fatalf("[%s] incorrect min read. expected %v, got %v", ext, 3, p.ReadingTime(context.Background()))
}
2014-01-29 17:50:31 -05:00
}
testAllMarkdownEnginesForPages(t, assertFunc, nil, simplePageWithLongContent)
2013-10-15 09:15:52 -04:00
}
2015-05-31 12:54:50 -04:00
func TestPagePaths(t *testing.T) {
t.Parallel()
c := qt.New(t)
siteParmalinksSetting := map[string]string{
"post": ":year/:month/:day/:title/",
}
tests := []struct {
content string
path string
hasPermalink bool
expected string
}{
{simplePage, "post/x.md", false, "post/x.html"},
{simplePageWithURL, "post/x.md", false, "simple/url/index.html"},
{simplePageWithSlug, "post/x.md", false, "post/simple-slug.html"},
{simplePageWithDate, "post/x.md", true, "2013/10/15/simple/index.html"},
{UTF8Page, "post/x.md", false, "post/x.html"},
{UTF8PageWithURL, "post/x.md", false, "ラーメン/url/index.html"},
{UTF8PageWithSlug, "post/x.md", false, "post/ラーメン-slug.html"},
{UTF8PageWithDate, "post/x.md", true, "2013/10/15/ラーメン/index.html"},
}
for _, test := range tests {
cfg, fs := newTestCfg()
configs, err := loadTestConfigFromProvider(cfg)
c.Assert(err, qt.IsNil)
if test.hasPermalink {
cfg.Set("permalinks", siteParmalinksSetting)
}
writeSource(t, fs, filepath.Join("content", filepath.FromSlash(test.path)), test.content)
s := buildSingleSite(t, deps.DepsCfg{Fs: fs, Configs: configs}, BuildCfg{SkipRender: true})
c.Assert(len(s.RegularPages()), qt.Equals, 1)
2015-08-30 18:51:25 -04:00
}
}
func TestTranslationKey(t *testing.T) {
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
files := `
-- hugo.toml --
disableKinds = ["taxonomy", "term"]
defaultContentLanguage = "en"
defaultContentLanguageInSubdir = true
[languages]
[languages.en]
weight = 1
[languages.nn]
weight = 2
-- content/sect/p1.en.md --
---
translationkey: "adfasdf"
title: "p1 en"
---
-- content/sect/p1.nn.md --
---
translationkey: "adfasdf"
title: "p1 nn"
---
-- layouts/_default/single.html --
Title: {{ .Title }}|TranslationKey: {{ .TranslationKey }}|
Translations: {{ range .Translations }}{{ .Language.Lang }}|{{ end }}|
AllTranslations: {{ range .AllTranslations }}{{ .Language.Lang }}|{{ end }}|
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
`
b := Test(t, files)
b.AssertFileContent("public/en/sect/p1/index.html",
"TranslationKey: adfasdf|",
"AllTranslations: en|nn||",
"Translations: nn||",
)
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
b.AssertFileContent("public/nn/sect/p1/index.html",
"TranslationKey: adfasdf|",
"Translations: en||",
"AllTranslations: en|nn||",
)
}
func TestTranslationKeyTermPages(t *testing.T) {
t.Parallel()
files := `
-- hugo.toml --
disableKinds = ['home','rss','section','sitemap','taxonomy']
defaultContentLanguage = 'en'
defaultContentLanguageInSubdir = true
[languages.en]
weight = 1
[languages.pt]
weight = 2
[taxonomies]
category = 'categories'
-- layouts/_default/list.html --
{{ .IsTranslated }}|{{ range .Translations }}{{ .RelPermalink }}|{{ end }}
-- layouts/_default/single.html --
{{ .Title }}|
-- content/p1.en.md --
---
title: p1 (en)
categories: [music]
---
-- content/p1.pt.md --
---
title: p1 (pt)
categories: [música]
---
-- content/categories/music/_index.en.md --
---
title: music
translationKey: foo
---
-- content/categories/música/_index.pt.md --
---
title: música
translationKey: foo
---
`
b := Test(t, files)
b.AssertFileContent("public/en/categories/music/index.html", "true|/pt/categories/m%C3%BAsica/|")
b.AssertFileContent("public/pt/categories/música/index.html", "true|/en/categories/music/|")
}
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
// Issue #11540.
func TestTranslationKeyResourceSharing(t *testing.T) {
files := `
-- hugo.toml --
disableKinds = ["taxonomy", "term"]
defaultContentLanguage = "en"
defaultContentLanguageInSubdir = true
[languages]
[languages.en]
weight = 1
[languages.nn]
weight = 2
-- content/sect/mybundle_en/index.en.md --
---
translationkey: "adfasdf"
title: "mybundle en"
---
-- content/sect/mybundle_en/f1.txt --
f1.en
-- content/sect/mybundle_en/f2.txt --
f2.en
-- content/sect/mybundle_nn/index.nn.md --
---
translationkey: "adfasdf"
title: "mybundle nn"
---
-- content/sect/mybundle_nn/f2.nn.txt --
f2.nn
-- layouts/_default/single.html --
Title: {{ .Title }}|TranslationKey: {{ .TranslationKey }}|
Resources: {{ range .Resources }}{{ .RelPermalink }}|{{ .Content }}|{{ end }}|
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
`
b := Test(t, files)
b.AssertFileContent("public/en/sect/mybundle_en/index.html",
"TranslationKey: adfasdf|",
"Resources: /en/sect/mybundle_en/f1.txt|f1.en|/en/sect/mybundle_en/f2.txt|f2.en||",
)
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
b.AssertFileContent("public/nn/sect/mybundle_nn/index.html",
"TranslationKey: adfasdf|",
"Title: mybundle nn|TranslationKey: adfasdf|\nResources: /en/sect/mybundle_en/f1.txt|f1.en|/nn/sect/mybundle_nn/f2.nn.txt|f2.nn||",
)
}
func TestChompBOM(t *testing.T) {
t.Parallel()
c := qt.New(t)
const utf8BOM = "\xef\xbb\xbf"
cfg, fs := newTestCfg()
configs, err := loadTestConfigFromProvider(cfg)
c.Assert(err, qt.IsNil)
writeSource(t, fs, filepath.Join("content", "simple.md"), utf8BOM+simplePage)
s := buildSingleSite(t, deps.DepsCfg{Fs: fs, Configs: configs}, BuildCfg{SkipRender: true})
c.Assert(len(s.RegularPages()), qt.Equals, 1)
p := s.RegularPages()[0]
checkPageTitle(t, p, "Simple")
}
func TestPageHTMLContent(t *testing.T) {
b := newTestSitesBuilder(t)
b.WithSimpleConfigFile()
frontmatter := `---
title: "HTML Content"
---
`
b.WithContent("regular.html", frontmatter+`<h1>Hugo</h1>`)
b.WithContent("nomarkdownforyou.html", frontmatter+`**Hugo!**`)
b.WithContent("manualsummary.html", frontmatter+`
<p>This is summary</p>
<!--more-->
<p>This is the main content.</p>`)
b.Build(BuildCfg{})
b.AssertFileContent(
"public/regular/index.html",
"Single: HTML Content|Hello|en|RelPermalink: /regular/|",
"Summary: Hugo|Truncated: false")
b.AssertFileContent(
"public/nomarkdownforyou/index.html",
"Permalink: http://example.com/nomarkdownforyou/|**Hugo!**|",
)
// https://github.com/gohugoio/hugo/issues/5723
b.AssertFileContent(
"public/manualsummary/index.html",
"Single: HTML Content|Hello|en|RelPermalink: /manualsummary/|",
"Summary: \n<p>This is summary</p>\n|Truncated: true",
"|<p>This is the main content.</p>|",
)
}
// https://github.com/gohugoio/hugo/issues/5381
func TestPageManualSummary(t *testing.T) {
b := newTestSitesBuilder(t)
b.WithSimpleConfigFile()
b.WithContent("page-md-shortcode.md", `---
title: "Hugo"
---
This is a {{< sc >}}.
<!--more-->
Content.
`)
// https://github.com/gohugoio/hugo/issues/5464
b.WithContent("page-md-only-shortcode.md", `---
title: "Hugo"
---
{{< sc >}}
<!--more-->
{{< sc >}}
`)
b.WithContent("page-md-shortcode-same-line.md", `---
title: "Hugo"
---
This is a {{< sc >}}<!--more-->Same line.
`)
b.WithContent("page-md-shortcode-same-line-after.md", `---
title: "Hugo"
---
Summary<!--more-->{{< sc >}}
`)
b.WithContent("page-org-shortcode.org", `#+TITLE: T1
#+AUTHOR: A1
#+DESCRIPTION: D1
This is a {{< sc >}}.
# more
Content.
`)
b.WithContent("page-org-variant1.org", `#+TITLE: T1
Summary.
# more
Content.
`)
b.WithTemplatesAdded("layouts/shortcodes/sc.html", "a shortcode")
b.WithTemplatesAdded("layouts/_default/single.html", `
SUMMARY:{{ .Summary }}:END
--------------------------
CONTENT:{{ .Content }}
`)
b.CreateSites().Build(BuildCfg{})
b.AssertFileContent("public/page-md-shortcode/index.html",
"SUMMARY:<p>This is a a shortcode.</p>:END",
"CONTENT:<p>This is a a shortcode.</p>\n\n<p>Content.</p>\n",
)
b.AssertFileContent("public/page-md-shortcode-same-line/index.html",
"SUMMARY:<p>This is a a shortcode</p>:END",
"CONTENT:<p>This is a a shortcode</p>\n\n<p>Same line.</p>\n",
)
b.AssertFileContent("public/page-md-shortcode-same-line-after/index.html",
"SUMMARY:<p>Summary</p>:END",
"CONTENT:<p>Summary</p>\n\na shortcode",
)
b.AssertFileContent("public/page-org-shortcode/index.html",
"SUMMARY:<p>\nThis is a a shortcode.\n</p>:END",
"CONTENT:<p>\nThis is a a shortcode.\n</p>\n<p>\nContent.\t\n</p>\n",
)
b.AssertFileContent("public/page-org-variant1/index.html",
"SUMMARY:<p>\nSummary.\n</p>:END",
"CONTENT:<p>\nSummary.\n</p>\n<p>\nContent.\t\n</p>\n",
)
b.AssertFileContent("public/page-md-only-shortcode/index.html",
"SUMMARY:a shortcode:END",
"CONTENT:a shortcode\n\na shortcode\n",
)
}
func TestHomePageWithNoTitle(t *testing.T) {
b := newTestSitesBuilder(t).WithConfigFile("toml", `
title = "Site Title"
`)
b.WithTemplatesAdded("index.html", "Title|{{ with .Title }}{{ . }}{{ end }}|")
b.WithContent("_index.md", `---
description: "No title for you!"
---
Content.
`)
b.Build(BuildCfg{})
b.AssertFileContent("public/index.html", "Title||")
}
func TestShouldBuild(t *testing.T) {
past := time.Date(2009, 11, 17, 20, 34, 58, 651387237, time.UTC)
future := time.Date(2037, 11, 17, 20, 34, 58, 651387237, time.UTC)
zero := time.Time{}
publishSettings := []struct {
buildFuture bool
buildExpired bool
buildDrafts bool
draft bool
publishDate time.Time
expiryDate time.Time
out bool
}{
// publishDate and expiryDate
{false, false, false, false, zero, zero, true},
{false, false, false, false, zero, future, true},
{false, false, false, false, past, zero, true},
{false, false, false, false, past, future, true},
{false, false, false, false, past, past, false},
{false, false, false, false, future, future, false},
{false, false, false, false, future, past, false},
// buildFuture and buildExpired
{false, true, false, false, past, past, true},
{true, true, false, false, past, past, true},
{true, false, false, false, past, past, false},
{true, false, false, false, future, future, true},
{true, true, false, false, future, future, true},
{false, true, false, false, future, past, false},
// buildDrafts and draft
{true, true, false, true, past, future, false},
{true, true, true, true, past, future, true},
{true, true, true, true, past, future, true},
}
for _, ps := range publishSettings {
s := shouldBuild(ps.buildFuture, ps.buildExpired, ps.buildDrafts, ps.draft,
ps.publishDate, ps.expiryDate)
if s != ps.out {
t.Errorf("AssertShouldBuild unexpected output with params: %+v", ps)
}
}
}
2017-01-02 05:44:17 -05:00
2022-04-26 13:57:04 -04:00
func TestShouldBuildWithClock(t *testing.T) {
htime.Clock = clocks.Start(time.Date(2021, 11, 17, 20, 34, 58, 651387237, time.UTC))
t.Cleanup(func() { htime.Clock = clocks.System() })
2022-04-26 13:57:04 -04:00
past := time.Date(2009, 11, 17, 20, 34, 58, 651387237, time.UTC)
future := time.Date(2037, 11, 17, 20, 34, 58, 651387237, time.UTC)
zero := time.Time{}
publishSettings := []struct {
buildFuture bool
buildExpired bool
buildDrafts bool
draft bool
publishDate time.Time
expiryDate time.Time
out bool
}{
// publishDate and expiryDate
{false, false, false, false, zero, zero, true},
{false, false, false, false, zero, future, true},
{false, false, false, false, past, zero, true},
{false, false, false, false, past, future, true},
{false, false, false, false, past, past, false},
{false, false, false, false, future, future, false},
{false, false, false, false, future, past, false},
// buildFuture and buildExpired
{false, true, false, false, past, past, true},
{true, true, false, false, past, past, true},
{true, false, false, false, past, past, false},
{true, false, false, false, future, future, true},
{true, true, false, false, future, future, true},
{false, true, false, false, future, past, false},
// buildDrafts and draft
{true, true, false, true, past, future, false},
{true, true, true, true, past, future, true},
{true, true, true, true, past, future, true},
}
for _, ps := range publishSettings {
s := shouldBuild(ps.buildFuture, ps.buildExpired, ps.buildDrafts, ps.draft,
ps.publishDate, ps.expiryDate)
if s != ps.out {
t.Errorf("AssertShouldBuildWithClock unexpected output with params: %+v", ps)
}
}
}
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
// See https://github.com/gohugoio/hugo/issues/9171
// We redefined disablePathToLower in v0.121.0.
func TestPagePathDisablePathToLower(t *testing.T) {
files := `
-- hugo.toml --
baseURL = "http://example.com"
disablePathToLower = true
[permalinks]
sect2 = "/:section/:filename/"
sect3 = "/:section/:title/"
-- content/sect/p1.md --
---
title: "Page1"
---
p1.
-- content/sect/p2.md --
---
title: "Page2"
slug: "PaGe2"
---
p2.
-- content/sect2/PaGe3.md --
---
title: "Page3"
---
-- content/seCt3/p4.md --
---
title: "Pag.E4"
slug: "PaGe4"
---
p4.
-- layouts/_default/single.html --
Single: {{ .Title}}|{{ .RelPermalink }}|{{ .Path }}|
`
b := Test(t, files)
b.AssertFileContent("public/sect/p1/index.html", "Single: Page1|/sect/p1/|/sect/p1")
b.AssertFileContent("public/sect/PaGe2/index.html", "Single: Page2|/sect/PaGe2/|/sect/p2")
b.AssertFileContent("public/sect2/PaGe3/index.html", "Single: Page3|/sect2/PaGe3/|/sect2/page3|")
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
b.AssertFileContent("public/sect3/Pag.E4/index.html", "Single: Pag.E4|/sect3/Pag.E4/|/sect3/p4|")
}
// https://github.com/gohugoio/hugo/issues/4675
func TestWordCountAndSimilarVsSummary(t *testing.T) {
t.Parallel()
c := qt.New(t)
single := []string{"_default/single.html", `
WordCount: {{ .WordCount }}
FuzzyWordCount: {{ .FuzzyWordCount }}
ReadingTime: {{ .ReadingTime }}
Len Plain: {{ len .Plain }}
Len PlainWords: {{ len .PlainWords }}
Truncated: {{ .Truncated }}
Len Summary: {{ len .Summary }}
Len Content: {{ len .Content }}
SUMMARY:{{ .Summary }}:{{ len .Summary }}:END
`}
b := newTestSitesBuilder(t)
b.WithSimpleConfigFile().WithTemplatesAdded(single...).WithContent("p1.md", fmt.Sprintf(`---
title: p1
---
%s
`, strings.Repeat("word ", 510)),
"p2.md", fmt.Sprintf(`---
title: p2
---
This is a summary.
<!--more-->
%s
`, strings.Repeat("word ", 310)),
"p3.md", fmt.Sprintf(`---
title: p3
isCJKLanguage: true
---
Summary: In Chinese, means good.
<!--more-->
%s
`, strings.Repeat("好", 200)),
"p4.md", fmt.Sprintf(`---
title: p4
isCJKLanguage: false
---
Summary: In Chinese, means good.
<!--more-->
%s
`, strings.Repeat("好", 200)),
"p5.md", fmt.Sprintf(`---
title: p4
isCJKLanguage: true
---
Summary: In Chinese, means good.
%s
`, strings.Repeat("好", 200)),
"p6.md", fmt.Sprintf(`---
title: p4
isCJKLanguage: false
---
Summary: In Chinese, means good.
%s
`, strings.Repeat("好", 200)),
)
b.CreateSites().Build(BuildCfg{})
c.Assert(len(b.H.Sites), qt.Equals, 1)
c.Assert(len(b.H.Sites[0].RegularPages()), qt.Equals, 6)
b.AssertFileContent("public/p1/index.html", "WordCount: 510\nFuzzyWordCount: 600\nReadingTime: 3\nLen Plain: 2550\nLen PlainWords: 510\nTruncated: false\nLen Summary: 2549\nLen Content: 2557")
b.AssertFileContent("public/p2/index.html", "WordCount: 314\nFuzzyWordCount: 400\nReadingTime: 2\nLen Plain: 1569\nLen PlainWords: 314\nTruncated: true\nLen Summary: 25\nLen Content: 1582")
b.AssertFileContent("public/p3/index.html", "WordCount: 206\nFuzzyWordCount: 300\nReadingTime: 1\nLen Plain: 638\nLen PlainWords: 7\nTruncated: true\nLen Summary: 43\nLen Content: 651")
b.AssertFileContent("public/p4/index.html", "WordCount: 7\nFuzzyWordCount: 100\nReadingTime: 1\nLen Plain: 638\nLen PlainWords: 7\nTruncated: true\nLen Summary: 43\nLen Content: 651")
b.AssertFileContent("public/p5/index.html", "WordCount: 206\nFuzzyWordCount: 300\nReadingTime: 1\nLen Plain: 638\nLen PlainWords: 7\nTruncated: true\nLen Summary: 229\nLen Content: 652")
b.AssertFileContent("public/p6/index.html", "WordCount: 7\nFuzzyWordCount: 100\nReadingTime: 1\nLen Plain: 638\nLen PlainWords: 7\nTruncated: false\nLen Summary: 637\nLen Content: 652")
}
2022-02-22 08:42:33 -05:00
func TestScratch(t *testing.T) {
t.Parallel()
b := newTestSitesBuilder(t)
b.WithSimpleConfigFile().WithTemplatesAdded("index.html", `
{{ .Scratch.Set "b" "bv" }}
B: {{ .Scratch.Get "b" }}
`,
"shortcodes/scratch.html", `
{{ .Scratch.Set "c" "cv" }}
C: {{ .Scratch.Get "c" }}
`,
)
b.WithContentAdded("scratchme.md", `
---
title: Scratch Me!
---
{{< scratch >}}
`)
b.Build(BuildCfg{})
b.AssertFileContent("public/index.html", "B: bv")
b.AssertFileContent("public/scratchme/index.html", "C: cv")
}
func TestPageParam(t *testing.T) {
t.Parallel()
b := newTestSitesBuilder(t).WithConfigFile("toml", `
baseURL = "https://example.org"
[params]
[params.author]
name = "Kurt Vonnegut"
`)
b.WithTemplatesAdded("index.html", `
{{ $withParam := .Site.GetPage "withparam" }}
{{ $noParam := .Site.GetPage "noparam" }}
{{ $withStringParam := .Site.GetPage "withstringparam" }}
Author page: {{ $withParam.Param "author.name" }}
Author name page string: {{ $withStringParam.Param "author.name" }}|
Author page string: {{ $withStringParam.Param "author" }}|
Author site config: {{ $noParam.Param "author.name" }}
`,
)
b.WithContent("withparam.md", `
+++
title = "With Param!"
[author]
name = "Ernest Miller Hemingway"
+++
`,
"noparam.md", `
---
title: "No Param!"
---
`, "withstringparam.md", `
+++
title = "With string Param!"
author = "Jo Nesbø"
+++
`)
b.Build(BuildCfg{})
b.AssertFileContent("public/index.html",
"Author page: Ernest Miller Hemingway",
"Author name page string: Kurt Vonnegut|",
"Author page string: Jo Nesbø|",
"Author site config: Kurt Vonnegut")
}
func TestGoldmark(t *testing.T) {
t.Parallel()
b := newTestSitesBuilder(t).WithConfigFile("toml", `
baseURL = "https://example.org"
[markup]
defaultMarkdownHandler="goldmark"
[markup.goldmark]
[markup.goldmark.renderer]
unsafe = false
[markup.highlight]
noClasses=false
`)
b.WithTemplatesAdded("_default/single.html", `
Title: {{ .Title }}
ToC: {{ .TableOfContents }}
Content: {{ .Content }}
`, "shortcodes/t.html", `T-SHORT`, "shortcodes/s.html", `## Code
{{ .Inner }}
`)
content := `
+++
title = "A Page!"
+++
## Shortcode {{% t %}} in header
## Code Fense in Shortcode
{{% s %}}
$$$bash {hl_lines=[1]}
SHORT
$$$
{{% /s %}}
## Code Fence
$$$bash {hl_lines=[1]}
MARKDOWN
$$$
Link with URL as text
[https://google.com](https://google.com)
`
content = strings.ReplaceAll(content, "$$$", "```")
b.WithContent("page.md", content)
b.Build(BuildCfg{})
b.AssertFileContent("public/page/index.html",
`<nav id="TableOfContents">
<li><a href="#shortcode-t-short-in-header">Shortcode T-SHORT in header</a></li>
<code class="language-bash" data-lang="bash"><span class="line hl"><span class="cl">SHORT
<code class="language-bash" data-lang="bash"><span class="line hl"><span class="cl">MARKDOWN
<p><a href="https://google.com">https://google.com</a></p>
`)
}
func TestPageHashString(t *testing.T) {
files := `
-- config.toml --
baseURL = "https://example.org"
[languages]
[languages.en]
weight = 1
title = "English"
[languages.no]
weight = 2
title = "Norsk"
-- content/p1.md --
---
title: "p1"
---
-- content/p2.md --
---
title: "p2"
---
`
b := NewIntegrationTestBuilder(IntegrationTestConfig{
T: t,
TxtarString: files,
}).Build()
p1 := b.H.Sites[0].RegularPages()[0]
p2 := b.H.Sites[0].RegularPages()[1]
sites := p1.Sites()
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning There are some breaking changes in this commit, see #11455. Closes #11455 Closes #11549 This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build. The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate. A list of the notable new features: * A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server. * A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently. * You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs. We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections). Memory Limit * Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory. New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively. This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive): Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers. Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds. Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role). Fixes #10169 Fixes #10364 Fixes #10482 Fixes #10630 Fixes #10656 Fixes #10694 Fixes #10918 Fixes #11262 Fixes #11439 Fixes #11453 Fixes #11457 Fixes #11466 Fixes #11540 Fixes #11551 Fixes #11556 Fixes #11654 Fixes #11661 Fixes #11663 Fixes #11664 Fixes #11669 Fixes #11671 Fixes #11807 Fixes #11808 Fixes #11809 Fixes #11815 Fixes #11840 Fixes #11853 Fixes #11860 Fixes #11883 Fixes #11904 Fixes #7388 Fixes #7425 Fixes #7436 Fixes #7544 Fixes #7882 Fixes #7960 Fixes #8255 Fixes #8307 Fixes #8863 Fixes #8927 Fixes #9192 Fixes #9324
2023-12-24 13:11:05 -05:00
b.Assert(p1, qt.Not(qt.Equals), p2)
b.Assert(identity.HashString(p1), qt.Not(qt.Equals), identity.HashString(p2))
b.Assert(identity.HashString(sites[0]), qt.Not(qt.Equals), identity.HashString(sites[1]))
}
// Issue #11243
func TestRenderWithoutArgument(t *testing.T) {
t.Parallel()
files := `
-- hugo.toml --
-- layouts/index.html --
{{ .Render }}
`
b, err := NewIntegrationTestBuilder(
IntegrationTestConfig{
T: t,
TxtarString: files,
},
).BuildE()
b.Assert(err, qt.IsNotNil)
}