There are some breaking changes in this commit, see #11455.
Closes#11455Closes#11549
This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build.
The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate.
A list of the notable new features:
* A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server.
* A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently.
* You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs.
We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections).
Memory Limit
* Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory.
New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively.
This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive):
Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get
Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers.
Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds.
Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role).
Fixes#10169Fixes#10364Fixes#10482Fixes#10630Fixes#10656Fixes#10694Fixes#10918Fixes#11262Fixes#11439Fixes#11453Fixes#11457Fixes#11466Fixes#11540Fixes#11551Fixes#11556Fixes#11654Fixes#11661Fixes#11663Fixes#11664Fixes#11669Fixes#11671Fixes#11807Fixes#11808Fixes#11809Fixes#11815Fixes#11840Fixes#11853Fixes#11860Fixes#11883Fixes#11904Fixes#7388Fixes#7425Fixes#7436Fixes#7544Fixes#7882Fixes#7960Fixes#8255Fixes#8307Fixes#8863Fixes#8927Fixes#9192Fixes#9324
The page images selection order as follows:
1. Page's images parameter, image resources are supported.
2. Page's image resources that naming in *feature*, *cover* or *thumbnail* pattern.
3. If no page images specified, then the first one of site's images will be used as the fallback, supports site resources.
This introduces a more automatic way of increasing the log levels for deprecation log statements based on the version it was deprecated.
The thresholds are a little arbitrary, but
* We log INFO for 6 releases
* We log WARN for another 6 releases
* THen ERROR (failing the build)
This should give theme authors plenty of time to catch up without having the log filled with warnings.
The template packages are based on go1.20.5 with the patch in befec5ddbbfbd81ec84e74e15a38044d67f8785b added.
This also includes a security fix that now disallows Go template actions in JS literals (inside backticks).
This will throw an error saying "... appears in a JS template literal".
If you're really sure this isn't a security risk in your case, you can revert to the old behaviour:
```toml
[security]
[security.gotemplates]
allowActionJSTmpl = true
```
See https://github.com/golang/go/issues/59234Fixes#11112
* Update math.Min and math.Max to allow 1 or more args, either scalar or slice, or combination of the two
* Add math.Sum (allow 1 or more args, either scalar or slice, or combination of the two)
* Add math.Product (allow 1 or more args, either scalar or slice, or combination of the two)
Fixes#11030
Primary motivation is documentation, but it will also hopefully simplify the code.
Also,
* Lower case the default output format names; this is in line with the custom ones (map keys) and how
it's treated all the places. This avoids doing `stringds.EqualFold` everywhere.
Closes#10896Closes#10620
strings.HasPrefix already has an alias of hasPrefix
but strings.HasSuffix has no such alias.
This PR adds a hasSuffix alias to the tpl function with corresponding
function documentation.
It also adds a Minor update to the hasPrefix function documentation
re: keywords and relatedfuncs.
Completes https://github.com/gohugoio/hugo/issues/10474
Make sure the context used for timeouts isn't created based on the incoming
context, as we have cases where this can cancel the context prematurely.
Fixes#10789
Before this fix, strings.Truncate would erroneously re-include
attributes from the opening tag in the closing tag when closing
formatted html, due to a bug in how tagnames were extracted from the
regex capture group for html tags used in `truncate.go`. This change
ensures that only the tagname is retained and all attributes are discarded
when storing the tags for closing them later.
Fixes#10399
Instead of maintaing a list of all CSS units and functions this commit:
* Uses 3 regexps to detect typed CSS values (e.g. `24px`) + properly handle numeric Go types.
* These regexps may have some false positives -- e.g. strings that needs to be quoted.
* For that rare case, you can mark the string with e.g. `"32xxx" | css.Quoted`
* For the opposite case: `"32" | css.Unquoted`
Updates #10632
The main topic of this commit is that you can now index fragments (content heading identifiers) when calling `.Related`.
You can do this by:
* Configure one or more indices with type `fragments`
* The name of those index configurations maps to an (optional) front matter slice with fragment references. This allows you to link
page<->fragment and page<->page.
* This also will index all the fragments (heading identifiers) of the pages.
It's also possible to use type `fragments` indices in shortcode, e.g.:
```
{{ $related := site.RegularPages.Related .Page }}
```
But, and this is important, you need to include the shortcode using the `{{<` delimiter. Not doing so will create infinite loops and timeouts.
This commit also:
* Adds two new methods to Page: Fragments (can also be used to build ToC) and HeadingsFiltered (this is only used in Related Content with
index type `fragments` and `enableFilter` set to true.
* Consolidates all `.Related*` methods into one, which takes either a `Page` or an options map as its only argument.
* Add `context.Context` to all of the content related Page API. Turns out it wasn't strictly needed for this particular feature, but it will
soon become usefil, e.g. in #9339.
Closes#10711
Updates #9339
Updates #10725
Note that this is backed by a LRU cache (which we soon shall see more usage of), so if you're a heavy user of cached partials it may be evicted and
refreshed if needed. But in most cases every partial is only invoked once.
This commit also adds a timeout (the global `timeout` config option) to make infinite recursion in partials
easier to reason about.
```
name old time/op new time/op delta
IncludeCached-10 8.92ms ± 0% 8.48ms ± 1% -4.87% (p=0.016 n=4+5)
name old alloc/op new alloc/op delta
IncludeCached-10 6.65MB ± 0% 5.17MB ± 0% -22.32% (p=0.002 n=6+6)
name old allocs/op new allocs/op delta
IncludeCached-10 117k ± 0% 71k ± 0% -39.44% (p=0.002 n=6+6)
```
Closes#4086
Updates #9588
When sorting strings a worng order is returned. This happens because the strings are first converted
to floating values to check whether or not they should be sorted as
floating values. When an error is returned the strings will be
handled as string literals.
No error will be returned when parsing Inf, Infinity or NaN (case insensitive) because they
will be coverted to special floating point values and therefore are
legal float values.
Now we check if the returned converted values are special floating
values and treat them as string literals.
Fixes#10389
