Commit graph

43 commits

Author SHA1 Message Date
Christian Oliff
17765a7451
all: Typo fixes 2024-04-11 09:23:17 +02:00
Bjørn Erik Pedersen
9df7b295bc
Filter dot files etc. in i18n
Closes #11993
2024-02-05 14:54:02 +01:00
Bjørn Erik Pedersen
982d9513e7
testing: Simplify some integration tests 2024-01-28 22:17:22 +01:00
Bjørn Erik Pedersen
2a0329423c
testing: Rename integration_test.go to PACKAGE_integration_test.go
Primary motivation making them easier to find in the code editor.
2024-01-28 11:41:59 +01:00
Bjørn Erik Pedersen
7285e74090
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning
There are some breaking changes in this commit, see #11455.

Closes #11455
Closes #11549

This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build.

The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate.

A list of the notable new features:

* A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server.
* A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently.
* You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs.
We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections).
Memory Limit
* Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory.

New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively.

This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive):

Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get
Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers.
Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds.

Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role).

Fixes #10169
Fixes #10364
Fixes #10482
Fixes #10630
Fixes #10656
Fixes #10694
Fixes #10918
Fixes #11262
Fixes #11439
Fixes #11453
Fixes #11457
Fixes #11466
Fixes #11540
Fixes #11551
Fixes #11556
Fixes #11654
Fixes #11661
Fixes #11663
Fixes #11664
Fixes #11669
Fixes #11671
Fixes #11807
Fixes #11808
Fixes #11809
Fixes #11815
Fixes #11840
Fixes #11853
Fixes #11860
Fixes #11883
Fixes #11904
Fixes #7388
Fixes #7425
Fixes #7436
Fixes #7544
Fixes #7882
Fixes #7960
Fixes #8255
Fixes #8307
Fixes #8863
Fixes #8927
Fixes #9192
Fixes #9324
2024-01-27 16:28:14 +01:00
Bjørn Erik Pedersen
ffd37d4f75 Only print the path warnings once
We could reset and rerun it on server rebuilds, but that report needs a full build to make sense.

Also clean up the config vs flags in this area: Make all config settings match the flags e.g. `printPathWarnings`, but set up aliases for the
old.

Fixes #11187
2023-06-30 10:24:28 +02:00
Bjørn Erik Pedersen
7c9fada778 Replace the old log setup, with structured logging etc.
Fixes #11124
2023-06-18 13:03:04 +02:00
Bjørn Erik Pedersen
7c7baa6183 Add hugo.WorkingDir
Fixes #10969
2023-05-20 17:45:56 +02:00
Oleksandr Redko
610cedaa61 all: Fix comments for exported functions and packages 2023-05-18 21:25:27 +02:00
Bjørn Erik Pedersen
241b21b0fd Create a struct with all of Hugo's config options
Primary motivation is documentation, but it will also hopefully simplify the code.

Also,

* Lower case the default output format names; this is in line with the custom ones (map keys) and how
it's treated all the places. This avoids doing `stringds.EqualFold` everywhere.

Closes #10896
Closes #10620
2023-05-16 18:01:29 +02:00
Joe Mooring
0cb6ca5906 langs/i18n: Fallback to defaultContentLanguage instead of English
Co-authored-by: 641bill <wo23636@126.com>

Fixes #9216
2023-04-24 21:20:09 +02:00
Bjørn Erik Pedersen
6c798eba60 Page context handling in i18n
This is a workaround. We need to improve on this, but not today.

Fixes #10782
2023-03-04 17:19:14 +01:00
Bjørn Erik Pedersen
fc9f315d86 Improve SASS errors
Fixes #9897
2022-05-15 20:25:25 +02:00
Bjørn Erik Pedersen
4b189d8fd9 postcss: Fix import error handling
Note that we will now fail if `inlineImports` is enabled and we cannot resolve an import.

You can work around this by either:

* Use url imports or imports with media queries.
* Set `skipInlineImportsNotFound=true` in the options

Also get the argument order in the different NewFileError* funcs in line.

Fixes #9895
2022-05-15 20:25:25 +02:00
Bjørn Erik Pedersen
5c96bda70a
errors: Misc improvements
* Redo the server error template
* Always add the content file context if relevant
* Remove some now superflous error string matching
* Move the server error template to _server/error.html
* Add file context (with position) to codeblock render blocks
* Improve JS build errors

Fixes #9892
Fixes #9891
Fixes #9893
2022-05-14 13:40:56 +02:00
Bjørn Erik Pedersen
f2946da9e8 Improve error messages, esp. when the server is running
* Add file context to minifier errors when publishing
* Misc fixes (see issues)
* Allow custom server error template in layouts/server/error.html

To get to this, this commit also cleans up and simplifies the code surrounding errors and files. This also removes the usage of `github.com/pkg/errors`, mostly because of https://github.com/pkg/errors/issues/223 -- but also because most of this is now built-in to Go.

Fixes #9852
Fixes #9857
Fixes #9863
2022-05-06 19:43:22 +02:00
Bjørn Erik Pedersen
30c2e54c25 Replace all usage of CopyOnWriteFs with OverlayFs
Fixes #9761
2022-04-10 13:49:31 +02:00
Bjørn Erik Pedersen
d070bdf10f
Rework the Destination filesystem to make --renderStaticToDisk work
See #9626
2022-04-08 13:26:17 +02:00
Bjørn Erik Pedersen
b80853de90
all: gofmt -w -r 'interface{} -> any' .
Updates #9687
2022-03-17 22:03:27 +01:00
Bjørn Erik Pedersen
4576c82ed4 Cache reflect.MethodByName
The isolated benchmark for the function is obviously much faster:

```bash
name                old time/op    new time/op    delta
GetMethodByName-10    1.21µs ± 7%    0.23µs ± 5%   -81.42%  (p=0.029 n=4+4)

name                old alloc/op   new alloc/op   delta
GetMethodByName-10      680B ± 0%        0B       -100.00%  (p=0.029 n=4+4)

name                old allocs/op  new allocs/op  delta
GetMethodByName-10      20.0 ± 0%       0.0       -100.00%  (p=0.029 n=4+4)
```

But more pleasing is the overall performance looking at the site benchmarks:

```bash
name                                      old time/op    new time/op    delta
SiteNew/Regular_Bundle_with_image-10        6.25ms ± 2%    6.10ms ± 2%     ~     (p=0.057 n=4+4)
SiteNew/Regular_Bundle_with_JSON_file-10    6.30ms ± 2%    5.66ms ±11%     ~     (p=0.057 n=4+4)
SiteNew/Regular_Tags_and_categories-10      22.2ms ± 2%    17.4ms ± 1%  -21.88%  (p=0.029 n=4+4)
SiteNew/Regular_Canonify_URLs-10             108ms ± 0%     107ms ± 0%   -1.20%  (p=0.029 n=4+4)
SiteNew/Regular_Deep_content_tree-10        36.1ms ± 1%    33.8ms ± 1%   -6.44%  (p=0.029 n=4+4)
SiteNew/Regular_TOML_front_matter-10        24.9ms ± 1%    22.6ms ± 1%   -9.30%  (p=0.029 n=4+4)
SiteNew/Regular_Many_HTML_templates-10      17.9ms ± 1%    16.7ms ± 1%   -6.43%  (p=0.029 n=4+4)
SiteNew/Regular_Page_collections-10         23.3ms ± 1%    22.0ms ± 0%   -5.58%  (p=0.029 n=4+4)
SiteNew/Regular_List_terms-10               8.00ms ± 1%    7.63ms ± 0%   -4.62%  (p=0.029 n=4+4)

name                                      old alloc/op   new alloc/op   delta
SiteNew/Regular_Bundle_with_image-10        2.10MB ± 0%    2.07MB ± 0%   -1.46%  (p=0.029 n=4+4)
SiteNew/Regular_Bundle_with_JSON_file-10    1.88MB ± 0%    1.85MB ± 0%   -1.76%  (p=0.029 n=4+4)
SiteNew/Regular_Tags_and_categories-10      13.5MB ± 0%    11.6MB ± 0%  -13.99%  (p=0.029 n=4+4)
SiteNew/Regular_Canonify_URLs-10            96.1MB ± 0%    95.8MB ± 0%   -0.40%  (p=0.029 n=4+4)
SiteNew/Regular_Deep_content_tree-10        28.4MB ± 0%    27.3MB ± 0%   -3.83%  (p=0.029 n=4+4)
SiteNew/Regular_TOML_front_matter-10        16.9MB ± 0%    15.1MB ± 0%  -10.58%  (p=0.029 n=4+4)
SiteNew/Regular_Many_HTML_templates-10      8.98MB ± 0%    8.44MB ± 0%   -6.04%  (p=0.029 n=4+4)
SiteNew/Regular_Page_collections-10         17.1MB ± 0%    16.5MB ± 0%   -3.91%  (p=0.029 n=4+4)
SiteNew/Regular_List_terms-10               3.92MB ± 0%    3.72MB ± 0%   -5.03%  (p=0.029 n=4+4)

name                                      old allocs/op  new allocs/op  delta
SiteNew/Regular_Bundle_with_image-10         25.8k ± 0%     24.9k ± 0%   -3.49%  (p=0.029 n=4+4)
SiteNew/Regular_Bundle_with_JSON_file-10     25.8k ± 0%     24.9k ± 0%   -3.49%  (p=0.029 n=4+4)
SiteNew/Regular_Tags_and_categories-10        288k ± 0%      233k ± 0%  -18.90%  (p=0.029 n=4+4)
SiteNew/Regular_Canonify_URLs-10              375k ± 0%      364k ± 0%   -2.80%  (p=0.029 n=4+4)
SiteNew/Regular_Deep_content_tree-10          314k ± 0%      283k ± 0%   -9.77%  (p=0.029 n=4+4)
SiteNew/Regular_TOML_front_matter-10          302k ± 0%      252k ± 0%  -16.55%  (p=0.029 n=4+4)
SiteNew/Regular_Many_HTML_templates-10        133k ± 0%      117k ± 0%  -11.81%  (p=0.029 n=4+4)
SiteNew/Regular_Page_collections-10           202k ± 0%      183k ± 0%   -9.55%  (p=0.029 n=4+4)
SiteNew/Regular_List_terms-10                55.6k ± 0%     49.8k ± 0%  -10.40%  (p=0.029 n=4+4)
```

Thanks to @quasilyte for the suggestion.

Fixes 9386
2022-03-08 19:36:55 +01:00
Joe Mooring
58adbeef88 Validate private use language tags
Fixes #9119
2021-12-02 16:30:53 +01:00
Bjørn Erik Pedersen
9185e11eff Reimplement archetypes
The old implementation had some issues, mostly related to the context (e.g. name, file paths) passed to the template.

This new implementation is using the exact same code path for evaluating the pages as in a regular build.

This also makes it more robust and easier to reason about in a multilingual setup.

Now, if you are explicit about the target path, Hugo will now always pick the correct mount and language:

```bash
hugo new content/en/posts/my-first-post.md
```

Fixes #9032
Fixes #7589
Fixes #9043
Fixes #9046
Fixes #9047
2021-10-16 15:22:03 +02:00
Bjørn Erik Pedersen
a3701e0931 Switch to go-toml v2
We have been using `go-toml` for language files only. This commit makes it the only TOML library.

It's spec compliant and very fast.

A benchark building a site with 200 pages with TOML front matter:

```bash
name                                  old time/op    new time/op    delta
SiteNew/Regular_TOML_front_matter-16    48.5ms ± 1%    47.1ms ± 1%  -2.85%  (p=0.029 n=4+4)

name                                  old alloc/op   new alloc/op   delta
SiteNew/Regular_TOML_front_matter-16    16.9MB ± 0%    16.7MB ± 0%  -1.56%  (p=0.029 n=4+4)

name                                  old allocs/op  new allocs/op  delta
SiteNew/Regular_TOML_front_matter-16      302k ± 0%      296k ± 0%  -2.20%  (p=0.029 n=4+4)
```

Note that the front matter unmarshaling is only a small part of building a site, so the above is very good.

Fixes #8801
2021-07-28 11:51:13 +02:00
Bjørn Erik Pedersen
022c479551
hugofs: Make FileMeta a struct
This commit started out investigating a `concurrent map read write` issue, ending by replacing the map with a struct.

This is easier to reason about, and it's more effective:

```
name                                  old time/op    new time/op    delta
SiteNew/Regular_Deep_content_tree-16    71.5ms ± 3%    69.4ms ± 5%    ~     (p=0.200 n=4+4)

name                                  old alloc/op   new alloc/op   delta
SiteNew/Regular_Deep_content_tree-16    29.7MB ± 0%    27.9MB ± 0%  -5.82%  (p=0.029 n=4+4)

name                                  old allocs/op  new allocs/op  delta
SiteNew/Regular_Deep_content_tree-16      313k ± 0%      303k ± 0%  -3.35%  (p=0.029 n=4+4)
```

See #8749
2021-07-15 17:14:26 +02:00
Bjørn Erik Pedersen
93aad3c543
Split out the puthe path/filepath functions into common/paths
So they can be used from the config package without cyclic troubles.

Updates #8654
2021-06-18 10:55:00 +02:00
Bjørn Erik Pedersen
d392893cd7
Misc config loading fixes
The main motivation behind this is simplicity and correctnes, but the new small config library is also faster:

```
BenchmarkDefaultConfigProvider/Viper-16         	  252418	      4546 ns/op	    2720 B/op	      30 allocs/op
BenchmarkDefaultConfigProvider/Custom-16        	  450756	      2651 ns/op	    1008 B/op	       6 allocs/op
```

Fixes #8633
Fixes #8618
Fixes #8630
Updates #8591
Closes #6680
Closes #5192
2021-06-14 17:00:32 +02:00
Bjørn Erik Pedersen
f55d2f4376
tpl/fmt: Add erroridf template func
Fixes #8613
2021-06-07 19:11:03 +02:00
Bjørn Erik Pedersen
ececd1b122 langs/i18n: Fix warning regression in i18n
Fix this by

1. Making sure that only numerical values are treated as plural counts
2. Making sure that `i18n.pluralFormNotFoundError` is not logged as a warning if `other` resolved.

Note that 2. isn't a new problem, but became visible because of the plural improvements in Hugo `0.83.0`.

Fixes #8492
2021-05-02 16:27:22 +02:00
Bjørn Erik Pedersen
7eb80a9e6f langs/i18n: Fix multiple unknown language codes
This fixes a bug upstream by temporarily using a patched version of go-i18n.

Closes #7838
2021-04-30 14:26:49 +02:00
Bjørn Erik Pedersen
eebde0c2ac langs/i18n: Improve plural handling of floats
The go-i18n library expects plural counts with floats to be represented as strings.

Fixes #8464
2021-04-25 11:12:30 +02:00
Bjørn Erik Pedersen
537c905ec1 langs/i18n: Revise the plural implementation
There were some issues introduced with the plural counting when we upgraded from v1 to v2 of go-i18n.

This commit improves that situation given the following rules:

* A single integer argument is used as plural count and passed to the i18n template as a int type with a `.Count` method. The latter is to preserve compability with v1.
* Else the plural count is either fetched from the `Count`/`count` field/method/map or from the value itself.
* Any data type is accepted, if it can be converted to an integer, that value is used.

The above means that you can now do pass a single integer and both of the below will work:

```
{{ . }} minutes to read
{{ .Count }} minutes to read
```

Fixes #8454
Closes #7822
See https://github.com/gohugoio/hugoDocs/issues/1410
2021-04-23 07:39:59 +02:00
Bjørn Erik Pedersen
92c6c40419
langs/i18n: Support translation files with suffix *.yml
Fixes #8212
2021-02-06 18:12:54 +01:00
Bjørn Erik Pedersen
d90e37e0c6 all: Format code with gofumpt
See https://github.com/mvdan/gofumpt
2020-12-03 13:12:58 +01:00
Bjørn Erik Pedersen
fdfa4a5fe6 Allow getJSON errors to be ignored
This change is mostly motivated to get a more stable CI build (we're building the Hugo site there, with Instagram and Twitter shortcodes sometimes failing).

Fixes #7866
2020-10-22 09:09:29 +02:00
Bjørn Erik Pedersen
506820435c lang/i18n: Fix for language code case issue with pt-br etc.
Fixes #7804
2020-10-09 18:02:43 +02:00
Bjørn Erik Pedersen
33e9d79b78 langs/i18n: Add workaround for known language, but missing plural rule error
Closes #7798
2020-10-08 14:20:18 +02:00
Bjørn Erik Pedersen
fc6abc39c7 langs/i18n: Fix for bare TOML keys
Hugo 0.76.0 updated go-i18n from v1 to v2. This allowed us to set the TOML unmarshaler to use, so we set the one we use in other places in Hugo.

But that does not support dotted bare keys, which caused some breakage in the wild.

This commit fixes that by:

* Using go-toml for language files
* Updating go-toml to the latest version
2020-10-08 14:20:18 +02:00
Bjørn Erik Pedersen
f9e798e8c4 langs/i18n: Fix i18n .Count regression
Fixes #7787
2020-10-07 00:32:21 +02:00
Bjørn Erik Pedersen
97987e5c02 langs/i18n: Upgrade to go-i18n v2
Fixes #5242
2020-09-29 17:48:07 +02:00
Bjørn Erik Pedersen
9e57182705
tests: Convert from testify to quicktest 2019-08-12 13:26:32 +02:00
Bjørn Erik Pedersen
00a238e32c Fix i18n project vs theme order
Fixes #6134
2019-07-29 19:10:15 +02:00
Bjørn Erik Pedersen
9f5a92078a
Add Hugo Modules
This commit implements Hugo Modules.

This is a broad subject, but some keywords include:

* A new `module` configuration section where you can import almost anything. You can configure both your own file mounts nd the file mounts of the modules you import. This is the new recommended way of configuring what you earlier put in `configDir`, `staticDir` etc. And it also allows you to mount folders in non-Hugo-projects, e.g. the `SCSS` folder in the Bootstrap GitHub project.
* A module consists of a set of mounts to the standard 7 component types in Hugo: `static`, `content`, `layouts`, `data`, `assets`, `i18n`, and `archetypes`. Yes, Theme Components can now include content, which should be very useful, especially in bigger multilingual projects.
* Modules not in your local file cache will be downloaded automatically and even "hot replaced" while the server is running.
* Hugo Modules supports and encourages semver versioned modules, and uses the minimal version selection algorithm to resolve versions.
* A new set of CLI commands are provided to manage all of this: `hugo mod init`,  `hugo mod get`,  `hugo mod graph`,  `hugo mod tidy`, and  `hugo mod vendor`.

All of the above is backed by Go Modules.

Fixes #5973
Fixes #5996
Fixes #6010
Fixes #5911
Fixes #5940
Fixes #6074
Fixes #6082
Fixes #6092
2019-07-24 09:35:53 +02:00
Bjørn Erik Pedersen
2838d58b1d
i18n: Move the package below /langs
To get fewer top level packages.
2019-05-04 18:25:56 +02:00