Commit graph

40 commits

Author SHA1 Message Date
Bjørn Erik Pedersen
982d9513e7
testing: Simplify some integration tests 2024-01-28 22:17:22 +01:00
Bjørn Erik Pedersen
2a0329423c
testing: Rename integration_test.go to PACKAGE_integration_test.go
Primary motivation making them easier to find in the code editor.
2024-01-28 11:41:59 +01:00
Bjørn Erik Pedersen
7285e74090
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning
There are some breaking changes in this commit, see #11455.

Closes #11455
Closes #11549

This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build.

The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate.

A list of the notable new features:

* A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server.
* A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently.
* You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs.
We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections).
Memory Limit
* Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory.

New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively.

This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive):

Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get
Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers.
Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds.

Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role).

Fixes #10169
Fixes #10364
Fixes #10482
Fixes #10630
Fixes #10656
Fixes #10694
Fixes #10918
Fixes #11262
Fixes #11439
Fixes #11453
Fixes #11457
Fixes #11466
Fixes #11540
Fixes #11551
Fixes #11556
Fixes #11654
Fixes #11661
Fixes #11663
Fixes #11664
Fixes #11669
Fixes #11671
Fixes #11807
Fixes #11808
Fixes #11809
Fixes #11815
Fixes #11840
Fixes #11853
Fixes #11860
Fixes #11883
Fixes #11904
Fixes #7388
Fixes #7425
Fixes #7436
Fixes #7544
Fixes #7882
Fixes #7960
Fixes #8255
Fixes #8307
Fixes #8863
Fixes #8927
Fixes #9192
Fixes #9324
2024-01-27 16:28:14 +01:00
Bjørn Erik Pedersen
8adba648cc all: Remove unused code
Using x/tools/cmd/deadcode
2023-12-18 19:51:26 +01:00
Joe Mooring
58da8554cb deps: Fix Chroma dependency version
Fixes #11311
2023-07-31 21:01:43 +02:00
Oleksandr Redko
382c726e63 markup: Fix typo in function and struct names 2023-06-01 17:59:44 +02:00
Oleksandr Redko
610cedaa61 all: Fix comments for exported functions and packages 2023-05-18 21:25:27 +02:00
Bjørn Erik Pedersen
241b21b0fd Create a struct with all of Hugo's config options
Primary motivation is documentation, but it will also hopefully simplify the code.

Also,

* Lower case the default output format names; this is in line with the custom ones (map keys) and how
it's treated all the places. This avoids doing `stringds.EqualFold` everywhere.

Closes #10896
Closes #10620
2023-05-16 18:01:29 +02:00
Bjørn Erik Pedersen
ce524d0b5e Add a page template func
Fixes #9339
2023-02-25 19:53:18 +01:00
Khayyam Saleem
73ece30d84
markup: Fix linenos codeblock hl option case regression
This fixes a regression introduced in v0.93.0 where previously, a
mixed-case key for lineNos would be successfully parsed. This change
moves the configuration key lowercasing step into the configuration
normalization stage, which is called whether the highlighting config
is being parsed from a `string` or a `map`.

Fixes #10682
2023-02-05 20:06:25 +01:00
Bjørn Erik Pedersen
e402d91ee1 Misc doc, code refactoring to improve documentation 2023-01-04 18:01:26 +01:00
Bjørn Erik Pedersen
7855b47f07 Add a cache for lexers.Get
```
name                            old time/op    new time/op    delta
Codeblocks/Default-10              152ms ±11%      12ms ± 1%  -92.44%  (p=0.029 n=4+4)
Codeblocks/Hook_no_higlight-10     142ms ± 0%       7ms ± 0%  -95.36%  (p=0.029 n=4+4)

name                            old alloc/op   new alloc/op   delta
Codeblocks/Default-10             11.9MB ± 0%    11.7MB ± 0%   -1.59%  (p=0.029 n=4+4)
Codeblocks/Hook_no_higlight-10    4.62MB ± 1%    4.43MB ± 0%   -4.08%  (p=0.029 n=4+4)

name                            old allocs/op  new allocs/op  delta
Codeblocks/Default-10               209k ± 0%      209k ± 0%   -0.03%  (p=0.029 n=4+4)
Codeblocks/Hook_no_higlight-10     68.4k ± 0%     68.3k ± 0%   -0.06%  (p=0.029 n=4+4)

```
2022-11-24 13:18:33 +01:00
Bjørn Erik Pedersen
bcb62d8911 deps: Upgrade github.com/alecthomas/chroma/v2 v2.4.0 2022-11-19 11:55:30 +01:00
Bjørn Erik Pedersen
d863dde6c6 markup/highlight: Add hl_inline option
Closes #9442
Closes #9635
Closes #9638
2022-06-15 20:04:20 +02:00
Bjørn Erik Pedersen
35fa192838 deps: Udpate to github.com/alecthomas/chroma/v2
Fixes #9932
Fixes #9931
2022-06-14 14:08:40 +02:00
Bjørn Erik Pedersen
b80853de90
all: gofmt -w -r 'interface{} -> any' .
Updates #9687
2022-03-17 22:03:27 +01:00
Bjørn Erik Pedersen
b82d95575d
Revert "markup/highlight: Add hl_inline option"
This reverts commit a360cab75a.
2022-03-10 10:06:22 +01:00
Bjørn Erik Pedersen
a360cab75a markup/highlight: Add hl_inline option
If set to true, the highlighted code will not be wrapped in any div.

Closes #9442
2022-03-09 11:33:13 +01:00
Bjørn Erik Pedersen
0327da050f tpl/transform: Fix it when template.HTML is passes as option to Hightlight
Fixes #9591
2022-03-02 12:30:35 +01:00
Bjørn Erik Pedersen
260ff1374d
markup/highlight: Ignore HL_lines_parsed in the gen docs 2022-02-28 08:33:34 +01:00
Bjørn Erik Pedersen
3ad39001df markup/highlight: Rework the return value from HighlightCodeblock
To make it possible to render it with a custom HTML ("<div>")  wrapper.

Updates #9573
2022-02-27 19:51:40 +01:00
Bjørn Erik Pedersen
f7109771a0 CodeblockContext method renames
Fixes #9577
2022-02-27 17:59:36 +01:00
Bjørn Erik Pedersen
579ff9b652 markup/goldmark: Improve attributes vs options
Fixes #9571
2022-02-26 21:54:36 +01:00
Bjørn Erik Pedersen
10928a4f78 Remove the trailing new line in .Code
Fixes #9572
2022-02-26 21:54:36 +01:00
Bjørn Erik Pedersen
0f80be341f markup/goldmark: Use Ordinal to create default lineanchors
The `Ordinal` starts at 0, so with a `hl-` prefix, this gives `hl-0-1` as a starting point.

Fixes #9567
2022-02-25 17:22:43 +01:00
Bjørn Erik Pedersen
08fdca9d93 Add Markdown diagrams and render hooks for code blocks
You can now create custom hook templates for code blocks, either one for all (`render-codeblock.html`) or for a given code language (e.g. `render-codeblock-go.html`).

We also used this new hook to add support for diagrams in Hugo:

* Goat (Go ASCII Tool) is built-in and enabled by default; just create a fenced code block with the language `goat` and start draw your Ascii diagrams.
* Another popular alternative for diagrams in Markdown, Mermaid (supported by GitHub), can also be implemented with a simple template. See the Hugo documentation for more information.

Updates #7765
Closes #9538
Fixes #9553
Fixes #8520
Fixes #6702
Fixes #9558
2022-02-24 18:59:50 +01:00
Bjørn Erik Pedersen
ea54a99ca5 deps: Update github.com/alecthomas/chroma v0.9.4 => v0.10.0 2022-02-15 10:37:14 +01:00
Helder Pereira
7a15edafe2 highlight: Add tabindex when code is not highlighted 2021-08-23 12:28:45 +02:00
rhymes
f27e542442
markup: Add tabindex="0" to default <pre> wrapper
Currently the generated `<pre>` element isn't fully accessible as it can't be focused by keyboard users.
To make this fully accessible, the attribute `tabindex="0"` should be added to the `<pre>` tag.

Closes #7194
2021-07-15 16:48:39 +02:00
Bjørn Erik Pedersen
d392893cd7
Misc config loading fixes
The main motivation behind this is simplicity and correctnes, but the new small config library is also faster:

```
BenchmarkDefaultConfigProvider/Viper-16         	  252418	      4546 ns/op	    2720 B/op	      30 allocs/op
BenchmarkDefaultConfigProvider/Custom-16        	  450756	      2651 ns/op	    1008 B/op	       6 allocs/op
```

Fixes #8633
Fixes #8618
Fixes #8630
Updates #8591
Closes #6680
Closes #5192
2021-06-14 17:00:32 +02:00
Phil Davis
04b89857e1
all: Fix minor typos 2020-12-16 12:11:32 +01:00
Bjørn Erik Pedersen
d90e37e0c6 all: Format code with gofumpt
See https://github.com/mvdan/gofumpt
2020-12-03 13:12:58 +01:00
Bjørn Erik Pedersen
b298c06e05
deps: Update to Chroma v0.8.2
Closes #7970
2020-11-20 09:30:05 +01:00
Cameron Moore
14bce18a6c highlight: Avoid making unnecessary allocation
Avoid creating a local copy of the highlight configuration when no
options are passed.

Benchmarks of building the docs site:

name        old time/op    new time/op    delta
DocsSite-2     1.94s ± 4%     1.93s ± 4%    ~     (p=0.841 n=5+5)

name        old alloc/op   new alloc/op   delta
DocsSite-2     666MB ± 1%     656MB ± 0%  -1.48%  (p=0.008 n=5+5)

name        old allocs/op  new allocs/op  delta
DocsSite-2     8.85M ± 0%     8.76M ± 0%  -1.04%  (p=0.029 n=4+4)
2020-10-16 21:14:02 +02:00
Fernando Jorge Mota
fb0f2cc718
markup/highlight: Add support to linkable line anchors on Chroma
Fixes #7622
2020-09-13 11:00:16 +02:00
satotake
3c568ad013
markup/highlight: Fix chroma highlight
* Use chroma.Coalesce
* Escape code strings if lexer is nil

Fixes #6877
Fixes #6856
2020-02-17 14:59:26 +01:00
Bjørn Erik Pedersen
40a092b068 markup: Reimplement pygmentsCodefencesGuessSyntax
Fixes #6565
2019-12-02 14:12:23 +01:00
Bjørn Erik Pedersen
a2d77f4a80
markup/highlight: Replace the temp for with a dependency 2019-11-24 13:59:16 +01:00
Cameron Moore
20f351ee4c Minor cleanups
Use fmt.Fprint were appropriate instead of Fprintf.  Remove some unused
code.
2019-11-23 16:38:02 +01:00
Bjørn Erik Pedersen
bfb9613a14
Add Goldmark as the new default markdown handler
This commit adds the fast and CommonMark compliant Goldmark as the new default markdown handler in Hugo.

If you want to continue using BlackFriday as the default for md/markdown extensions, you can use this configuration:

```toml
[markup]
defaultMarkdownHandler="blackfriday"
```

Fixes #5963
Fixes #1778
Fixes #6355
2019-11-23 14:12:24 +01:00