Commit graph

68 commits

Author SHA1 Message Date
Bjørn Erik Pedersen
8e50ccfae7 github: Add a "free space" step on Ubuntu 2024-04-14 16:42:04 +02:00
Bjørn Erik Pedersen
00ae8e8c72 github: Update actions 2024-04-14 13:40:46 +02:00
Bjørn Erik Pedersen
e423e56273 github: Format GitHub actions files 2024-04-14 13:40:46 +02:00
Bjørn Erik Pedersen
9323376dfa github: Fix CI build 2024-04-11 13:29:19 +02:00
Bjørn Erik Pedersen
9571246bcc Upgrade to Go 1.22
Fixes #12010
2024-02-07 20:42:42 +01:00
Bjørn Erik Pedersen
7285e74090
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning
There are some breaking changes in this commit, see #11455.

Closes #11455
Closes #11549

This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build.

The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate.

A list of the notable new features:

* A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server.
* A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently.
* You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs.
We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections).
Memory Limit
* Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory.

New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively.

This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive):

Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get
Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers.
Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds.

Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role).

Fixes #10169
Fixes #10364
Fixes #10482
Fixes #10630
Fixes #10656
Fixes #10694
Fixes #10918
Fixes #11262
Fixes #11439
Fixes #11453
Fixes #11457
Fixes #11466
Fixes #11540
Fixes #11551
Fixes #11556
Fixes #11654
Fixes #11661
Fixes #11663
Fixes #11664
Fixes #11669
Fixes #11671
Fixes #11807
Fixes #11808
Fixes #11809
Fixes #11815
Fixes #11840
Fixes #11853
Fixes #11860
Fixes #11883
Fixes #11904
Fixes #7388
Fixes #7425
Fixes #7436
Fixes #7544
Fixes #7882
Fixes #7960
Fixes #8255
Fixes #8307
Fixes #8863
Fixes #8927
Fixes #9192
Fixes #9324
2024-01-27 16:28:14 +01:00
Bjørn Erik Pedersen
8adba648cc all: Remove unused code
Using x/tools/cmd/deadcode
2023-12-18 19:51:26 +01:00
Bjørn Erik Pedersen
6d4b012412
github: Fix CI build on Windows 2023-11-30 19:22:29 +01:00
Joe Mooring
2e4bf89ec7 misc: Change dart-sass labels in workflows and snapcraft 2023-08-23 22:51:46 +02:00
Bjørn Erik Pedersen
24b1be45c1 Go 1.21 Upgrade
Fixes #11351
2023-08-23 21:49:27 +02:00
Joe Mooring
8fa8ce3e48 Update GitHub issue template 2023-08-02 18:07:31 +02:00
Bjørn Erik Pedersen
19d76ae96b github: Build for Dragonfly in CI build
To discover upstream issues before we release.
2023-07-01 18:53:48 +02:00
Bjørn Erik Pedersen
5b4bfc2db7 Fix broken nodeploy setup
Fixes #11149
2023-06-23 12:40:43 +02:00
Bjørn Erik Pedersen
f210188da3 Upgrade to v2 of the Dart Sass Embedded Protocol
Fixes #11059
2023-06-12 13:47:38 +02:00
Bjørn Erik Pedersen
06faee5beb github: Fix Windows build
Also update some Actions to get rid of some warnings.

Fixes #11052
2023-06-01 09:54:40 +02:00
Bjørn Erik Pedersen
35955f50e6
github: Trim the test flow a little
We now have much improved end to end tests, so cut the building of the docs site to save some time of the PR builds.
2023-05-17 22:14:15 +02:00
Bjørn Erik Pedersen
4801e2e8ee build: Update to Go 1.20
Updates #10691
2023-02-22 11:26:52 +01:00
Bjørn Erik Pedersen
c9354d5463 github: Update to Dart Sass 1.56.2 2022-12-13 16:58:33 +01:00
Bjørn Erik Pedersen
d894269855 github: Use ruby/setup-ruby
Closes #10517
2022-12-11 19:43:46 +01:00
Bjørn Erik Pedersen
e93138dfdb dartsass: Add sourceMapIncludeSources option 2022-12-05 16:36:45 +01:00
Bjørn Erik Pedersen
7d16c3c0c1 github: Update Dart Sass Embedded to 1.56.1 2022-12-05 16:36:45 +01:00
Bjørn Erik Pedersen
d1cd1db0e7
github: Avoid duplicate test runs 2022-10-25 16:08:41 +02:00
Bjørn Erik Pedersen
1fd3320dcf
github: Use SHA versions 2022-10-05 16:37:37 +02:00
jongwooo
0fb2b3d143 Resolve dependency-path not found error in workflow 2022-10-05 20:39:45 +09:00
jongwooo
db05232d56 Use setup-go action to cache dependencies 2022-10-05 10:20:22 +02:00
Alex
5e03de0ddb Update stale.yml
Signed-off-by: sashashura <93376818+sashashura@users.noreply.github.com>
2022-09-02 20:13:02 +02:00
Anthony Fok
f7e00c039f
github: Use GitHub's Choco-Install function to retry installs
Apparently, two instances of "choco install mingw" running simultaneously
(for go-version 1.18.x and 1.19.x) would often cause at least one of them
to fail to connect to SourceForge:

    ERROR: The remote file either doesn't exist, is unauthorized,
    or is forbidden for url
    'https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/ray_linn/GCC-10.X-with-ada/GCC-10.2.0-crt-8.0.0-with-ada-20201019.7z/download'.
    Exception calling "GetResponse" with "0" argument(s):
    "The request was aborted: Could not create SSL/TLS secure channel."

With Choco-Install however, it simply works every single time!

Thanks to keymanapp/api.keyman.com#156 for the great tip!

See the definition of the PowerShell Choco-Install function here:
https://github.com/actions/runner-images/blob/main/images/win/scripts/ImageHelpers/ChocoHelpers.ps1
2022-08-14 00:52:23 -06:00
Bjørn Erik Pedersen
0cd1929b9c Update to Go 1.19
Fixes #10145
2022-08-03 23:22:01 +02:00
Bjørn Erik Pedersen
223bf28004 parser/pageparser: Don't store the byte slices
On its own this change doesn't do any magic, but this is part of a bigger picture about making Hugo leaner in the
memory usage department.
2022-07-09 16:03:11 +02:00
Bjørn Erik Pedersen
52edea0fec github: Set HUGO_BUILD_TAGS: extended when running tests
Also fix TestDecodeConfig/Basic which started to fail in the extended build in 0.99.1.

Closes #9935
2022-05-27 11:55:03 +02:00
Bjørn Erik Pedersen
3a8189ee93
Update stale.yml 2022-05-17 13:57:01 +02:00
nathannaveen
a6d545854a
github: Add permissions to test action 2022-05-03 09:23:47 +02:00
Anthony Fok
bbd455fe79 Update CONTRIBUTING.md to use "go install" to install mage
In Go 1.18, "go get" no longer builds packages.
2022-03-21 21:01:24 -06:00
Bjørn Erik Pedersen
9d6495d774
github: Make it build with Go 1.18
I also ran some benchmarks comparing it to Go 1.17.8:

```bash
name                                      old time/op    new time/op    delta
SiteNew/Regular_Bundle_with_image-10        5.13ms ± 7%    4.23ms ± 7%  -17.47%  (p=0.029 n=4+4)
SiteNew/Regular_Bundle_with_JSON_file-10    5.12ms ± 1%    4.16ms ± 1%  -18.65%  (p=0.029 n=4+4)
SiteNew/Regular_Tags_and_categories-10      17.3ms ± 2%    15.1ms ± 1%  -12.53%  (p=0.029 n=4+4)
SiteNew/Regular_Canonify_URLs-10             106ms ± 1%      90ms ± 0%  -15.86%  (p=0.029 n=4+4)
SiteNew/Regular_Deep_content_tree-10        34.0ms ± 1%    30.0ms ± 2%  -11.72%  (p=0.029 n=4+4)
SiteNew/Regular_TOML_front_matter-10        22.7ms ± 2%    20.2ms ± 1%  -10.93%  (p=0.029 n=4+4)
SiteNew/Regular_Many_HTML_templates-10      16.7ms ± 0%    14.6ms ± 1%  -12.89%  (p=0.029 n=4+4)
SiteNew/Regular_Page_collections-10         22.2ms ± 1%    19.5ms ± 1%  -12.07%  (p=0.029 n=4+4)
SiteNew/Regular_List_terms-10               7.71ms ± 0%    6.59ms ± 1%  -14.52%  (p=0.029 n=4+4)

name                                      old alloc/op   new alloc/op   delta
SiteNew/Regular_Bundle_with_image-10        2.07MB ± 0%    2.04MB ± 0%   -1.40%  (p=0.029 n=4+4)
SiteNew/Regular_Bundle_with_JSON_file-10    1.85MB ± 0%    1.83MB ± 0%   -1.54%  (p=0.029 n=4+4)
SiteNew/Regular_Tags_and_categories-10      11.8MB ± 0%    11.1MB ± 0%   -6.11%  (p=0.029 n=4+4)
SiteNew/Regular_Canonify_URLs-10            95.8MB ± 0%    95.2MB ± 0%   -0.63%  (p=0.029 n=4+4)
SiteNew/Regular_Deep_content_tree-10        27.4MB ± 0%    26.9MB ± 0%   -1.68%  (p=0.029 n=4+4)
SiteNew/Regular_TOML_front_matter-10        15.2MB ± 0%    14.4MB ± 0%   -4.93%  (p=0.029 n=4+4)
SiteNew/Regular_Many_HTML_templates-10      8.45MB ± 0%    8.24MB ± 0%   -2.43%  (p=0.029 n=4+4)
SiteNew/Regular_Page_collections-10         16.5MB ± 0%    16.2MB ± 0%   -1.78%  (p=0.029 n=4+4)
SiteNew/Regular_List_terms-10               3.73MB ± 0%    3.64MB ± 0%   -2.30%  (p=0.029 n=4+4)

name                                      old allocs/op  new allocs/op  delta
SiteNew/Regular_Bundle_with_image-10         25.0k ± 0%     24.1k ± 0%   -3.60%  (p=0.029 n=4+4)
SiteNew/Regular_Bundle_with_JSON_file-10     25.1k ± 0%     24.1k ± 0%   -3.60%  (p=0.029 n=4+4)
SiteNew/Regular_Tags_and_categories-10        237k ± 0%      234k ± 0%   -1.29%  (p=0.029 n=4+4)
SiteNew/Regular_Canonify_URLs-10              365k ± 0%      363k ± 0%   -0.47%  (p=0.029 n=4+4)
SiteNew/Regular_Deep_content_tree-10          284k ± 0%      279k ± 0%   -1.98%  (p=0.029 n=4+4)
SiteNew/Regular_TOML_front_matter-10          252k ± 0%      249k ± 0%   -1.48%  (p=0.029 n=4+4)
SiteNew/Regular_Many_HTML_templates-10        117k ± 0%      115k ± 0%   -1.68%  (p=0.029 n=4+4)
SiteNew/Regular_Page_collections-10           184k ± 0%      181k ± 0%   -1.74%  (p=0.029 n=4+4)
SiteNew/Regular_List_terms-10                49.9k ± 0%     48.5k ± 0%   -2.79%  (p=0.029 n=4+4)
```
2022-03-16 11:32:39 +01:00
Bjørn Erik Pedersen
a6488e7bad
Remove Go 1.17 support
Go 1.18 had some breaking changes on source level meaning we cannot build on older Go versions anymore.

The improvements in Go 1.18 (e.g. `break` and `continue`)  were to good to pass on.

Note that you don't need Go (or Go 1.18) to run a pre-built binary.

Updates #9677
2022-03-16 08:57:58 +01:00
Bjørn Erik Pedersen
4d6d1d08da
build: Bump to Go 1.18
Updates #9677
2022-03-16 08:43:18 +01:00
Bjørn Erik Pedersen
19f816f77e
Update stale.yml
operations-per-run: 999

We need to get this done so the notification gets back to normal.
2022-03-08 09:09:38 +01:00
Bjørn Erik Pedersen
b0eea00756
Update stale.yml
Updates #9522
2022-02-18 11:43:35 +01:00
Bjørn Erik Pedersen
ef8b781b14
Update stale.yml
Updates #9522
2022-02-18 11:38:24 +01:00
Bjørn Erik Pedersen
3136ff67d8
github: Configure stale to be run manually
Updates #9522
2022-02-18 11:31:57 +01:00
Bjørn Erik Pedersen
6041adc164
github: Clean up the issue templates a little 2022-01-13 09:44:55 +01:00
Bjørn Erik Pedersen
408da43657
github: Add lock-threads step 2022-01-12 19:55:53 +01:00
Bjørn Erik Pedersen
85c5b89593
github: Increase stale days 2022-01-05 15:48:00 +01:00
Bjørn Erik Pedersen
dad0dc8d86
github: Add add stale GitHub action 2022-01-04 17:08:10 +01:00
Bjørn Erik Pedersen
672481f1f0
Update stale.yml 2021-12-30 20:13:08 +01:00
Bjørn Erik Pedersen
0fc2ce9e4b
Update to Go 1.17
Go 1.17 now lazy-loads dependencies when `go.mod` is version `go17`. This does not work for us for new projects started with `hugo mod init`.

Before this commit, starting a project with Go 1.17 with `hugo mod init` and then start adding dependencies with transitive dependenies to `config.toml` would treat the transitive dependencies as new, and you would potentially get a too recent version of those.

Note that this does not effect existing projects, where all dependencies are already recorded in `go.mod`.

Fixes #8930
2021-08-31 11:19:51 +02:00
Bjørn Erik Pedersen
a886dd53b8
github: Set a dummy Instagram token
For now, need to debug why this fails.
2021-06-13 11:09:50 +02:00
Bjørn Erik Pedersen
9b5debe4b8 Upgrade Instagram shortcode
Fixes #7879
2021-06-08 10:30:45 +02:00
Patrick Kollitsch
137d2dab32
github: More explicit support link to discourse 2021-03-14 18:27:15 +01:00
Bjørn Erik Pedersen
718fba7d63 Update Travis, GitHub, CircleCI and Snap to Go 1.16 (only) 2021-02-18 14:11:48 +01:00