Commit graph

89 commits

Author SHA1 Message Date
Bjørn Erik Pedersen
6add6d77b4 Rename transpileJS to babel
And add a test.

Updates #5764
2020-04-29 10:51:33 +02:00
Niek de Wit
2a171ff1c5 resources: Add JavaScript transpiling solution
Add a new pipe called TranspileJS which uses the Babel cli. This makes it possible for users to write ES6 JavaScript code and transpile it to ES5 during website generation so that the code still works with older browser versions.

Fixes #5764
2020-04-29 10:51:33 +02:00
Bjørn Erik Pedersen
2f721f8ec6
Add basic "post resource publish support"
Fixes #7146
2020-04-07 21:59:20 +02:00
Bjørn Erik Pedersen
523d51948f Fix _build.list.local logic
Fixes #7089
2020-03-24 12:47:42 +01:00
Bjørn Erik Pedersen
c9dc316ad1
Revert "resources: Add data context to the key in ExecuteAsTemplate"
This reverts commit 18cb21ff2e.

Fixes #7064
2020-03-21 18:38:19 +01:00
Bjørn Erik Pedersen
7204b354a9 Some minify configuration adjustments 2020-03-20 20:35:57 +01:00
SatowTakeshi
574c2959b8 Add minify config
Fixes #6750
Updates #6892
2020-03-20 20:35:57 +01:00
Bjørn Erik Pedersen
99958f90fe Allow headless bundles to list pages via $page.Pages and $page.RegularPages
Fixes #7075
2020-03-20 18:28:55 +01:00
Bjørn Erik Pedersen
c7b6d74e89
resources: Fix scss vs css import regexp
Fixes #7063
2020-03-17 08:03:25 +01:00
Bjørn Erik Pedersen
1a8af7d4f0 Add workaround for regular CSS imports in SCSS
Fixes #7059
2020-03-16 23:21:44 +01:00
Bjørn Erik Pedersen
03b93bb988 Add .RegularPagesRecursive
Fixes #6411
2020-03-16 17:37:36 +01:00
Bjørn Erik Pedersen
18cb21ff2e
resources: Add data context to the key in ExecuteAsTemplate
Fixes #7046
2020-03-11 22:21:16 +01:00
Bjørn Erik Pedersen
df298558a5
Improve Tailwind/PostCSS error messages
Fixes #7041
Fixes #7042
2020-03-11 14:13:03 +01:00
Bjørn Erik Pedersen
c0177fe2b2
resources: Try to fix a Go 1.15 go vet error 2020-03-10 12:29:09 +01:00
Bjørn Erik Pedersen
3d3fa5c3fe Add build.UseResourceCacheWhen
Fixes #6993
2020-03-03 13:29:58 +01:00
Bjørn Erik Pedersen
305ce1c9ec
resources: Add full filename to image when processing fails
Fixes #7000
2020-03-02 12:39:26 +01:00
Bjørn Erik Pedersen
b66d38c419 resources: Add basic @import support to resources.PostCSS
This commit also makes the HUGO_ENVIRONMENT environment variable available to Node.

Fixes #6957
Fixes #6961
2020-02-27 11:47:24 +01:00
Bjørn Erik Pedersen
76b2afe642
Apply missing go fmt 2020-02-21 10:12:33 +01:00
Bjørn Erik Pedersen
f460530347
Fix panic on no output formats
A page needs its output formats even if it should not be rendered or its resources should not be published.

Fixes #6924
2020-02-21 09:41:39 +01:00
Bjørn Erik Pedersen
fa520a2d98 Add Page.GetTerms
Fixes #6905
2020-02-19 14:52:23 +01:00
Bjørn Erik Pedersen
eada236f87
Introduce a tree map for all content
This commit introduces a new data structure to store pages and their resources.

This data structure is backed by radix trees.

This simplies tree operations, makes all pages a bundle,  and paves the way for #6310.

It also solves a set of annoying issues (see list below).

Not a motivation behind this, but this commit also makes Hugo in general a little bit faster and more memory effective (see benchmarks). Especially for partial rebuilds on content edits, but also when taxonomies is in use.

```
name                                   old time/op    new time/op    delta
SiteNew/Bundle_with_image/Edit-16        1.32ms ± 8%    1.00ms ± 9%  -24.42%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file/Edit-16    1.28ms ± 0%    0.94ms ± 0%  -26.26%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories/Edit-16      33.9ms ± 2%    21.8ms ± 1%  -35.67%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs/Edit-16            40.6ms ± 1%    37.7ms ± 3%   -7.20%  (p=0.029 n=4+4)
SiteNew/Deep_content_tree/Edit-16        56.7ms ± 0%    51.7ms ± 1%   -8.82%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates/Edit-16      19.9ms ± 2%    18.3ms ± 3%   -7.64%  (p=0.029 n=4+4)
SiteNew/Page_collections/Edit-16         37.9ms ± 4%    34.0ms ± 2%  -10.28%  (p=0.029 n=4+4)
SiteNew/Bundle_with_image-16             10.7ms ± 0%    10.6ms ± 0%   -1.15%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file-16         10.8ms ± 0%    10.7ms ± 0%   -1.05%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories-16           43.2ms ± 1%    39.6ms ± 1%   -8.35%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs-16                 47.6ms ± 1%    47.3ms ± 0%     ~     (p=0.057 n=4+4)
SiteNew/Deep_content_tree-16             73.0ms ± 1%    74.2ms ± 1%     ~     (p=0.114 n=4+4)
SiteNew/Many_HTML_templates-16           37.9ms ± 0%    38.1ms ± 1%     ~     (p=0.114 n=4+4)
SiteNew/Page_collections-16              53.6ms ± 1%    54.7ms ± 1%   +2.09%  (p=0.029 n=4+4)

name                                   old alloc/op   new alloc/op   delta
SiteNew/Bundle_with_image/Edit-16         486kB ± 0%     430kB ± 0%  -11.47%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file/Edit-16     265kB ± 0%     209kB ± 0%  -21.06%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories/Edit-16      13.6MB ± 0%     8.8MB ± 0%  -34.93%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs/Edit-16            66.5MB ± 0%    63.9MB ± 0%   -3.95%  (p=0.029 n=4+4)
SiteNew/Deep_content_tree/Edit-16        28.8MB ± 0%    25.8MB ± 0%  -10.55%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates/Edit-16      6.16MB ± 0%    5.56MB ± 0%   -9.86%  (p=0.029 n=4+4)
SiteNew/Page_collections/Edit-16         16.9MB ± 0%    16.0MB ± 0%   -5.19%  (p=0.029 n=4+4)
SiteNew/Bundle_with_image-16             2.28MB ± 0%    2.29MB ± 0%   +0.35%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file-16         2.07MB ± 0%    2.07MB ± 0%     ~     (p=0.114 n=4+4)
SiteNew/Tags_and_categories-16           14.3MB ± 0%    13.2MB ± 0%   -7.30%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs-16                 69.1MB ± 0%    69.0MB ± 0%     ~     (p=0.343 n=4+4)
SiteNew/Deep_content_tree-16             31.3MB ± 0%    31.8MB ± 0%   +1.49%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates-16           10.8MB ± 0%    10.9MB ± 0%   +1.11%  (p=0.029 n=4+4)
SiteNew/Page_collections-16              21.4MB ± 0%    21.6MB ± 0%   +1.15%  (p=0.029 n=4+4)

name                                   old allocs/op  new allocs/op  delta
SiteNew/Bundle_with_image/Edit-16         4.74k ± 0%     3.86k ± 0%  -18.57%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file/Edit-16     4.73k ± 0%     3.85k ± 0%  -18.58%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories/Edit-16        301k ± 0%      198k ± 0%  -34.14%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs/Edit-16              389k ± 0%      373k ± 0%   -4.07%  (p=0.029 n=4+4)
SiteNew/Deep_content_tree/Edit-16          338k ± 0%      262k ± 0%  -22.63%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates/Edit-16        102k ± 0%       88k ± 0%  -13.81%  (p=0.029 n=4+4)
SiteNew/Page_collections/Edit-16           176k ± 0%      152k ± 0%  -13.32%  (p=0.029 n=4+4)
SiteNew/Bundle_with_image-16              26.8k ± 0%     26.8k ± 0%   +0.05%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file-16          26.8k ± 0%     26.8k ± 0%   +0.05%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories-16             273k ± 0%      245k ± 0%  -10.36%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs-16                   396k ± 0%      398k ± 0%   +0.39%  (p=0.029 n=4+4)
SiteNew/Deep_content_tree-16               317k ± 0%      325k ± 0%   +2.53%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates-16             146k ± 0%      147k ± 0%   +0.98%  (p=0.029 n=4+4)
SiteNew/Page_collections-16                210k ± 0%      215k ± 0%   +2.44%  (p=0.029 n=4+4)
```

Fixes #6312
Fixes #6087
Fixes #6738
Fixes #6412
Fixes #6743
Fixes #6875
Fixes #6034
Fixes #6902
Fixes #6173
Fixes #6590
2020-02-18 09:49:42 +01:00
Bjørn Erik Pedersen
40ba7e6d63 Update to LibSass v3.6.3
Fixes #6862
2020-02-10 10:42:05 +01:00
Bjørn Erik Pedersen
c6d650c8c8
tpl/tplimpl: Rework template management to get rid of concurrency issues
This more or less completes the simplification of the template handling code in Hugo started in v0.62.

The main motivation was to fix a long lasting issue about a crash in HTML content files  without front matter.

But this commit also comes with a big functional improvement.

As we now have moved the base template evaluation to the build stage we now use the same lookup rules for `baseof` as for `list` etc. type of templates.

This means that in this simple example you can have a `baseof` template for the `blog` section without having to duplicate the others:

```
layouts
├── _default
│   ├── baseof.html
│   ├── list.html
│   └── single.html
└── blog
    └── baseof.html
```

Also, when simplifying code, you often get rid of some double work, as shown in the "site building" benchmarks below.

These benchmarks looks suspiciously good, but I have repeated the below with ca. the same result. Compared to master:

```
name                              old time/op    new time/op    delta
SiteNew/Bundle_with_image-16        13.1ms ± 1%    10.5ms ± 1%  -19.34%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file-16    13.0ms ± 0%    10.7ms ± 1%  -18.05%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories-16      46.4ms ± 2%    43.1ms ± 1%   -7.15%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs-16            52.2ms ± 2%    47.8ms ± 1%   -8.30%  (p=0.029 n=4+4)
SiteNew/Deep_content_tree-16        77.9ms ± 1%    70.9ms ± 1%   -9.01%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates-16      43.0ms ± 0%    37.2ms ± 1%  -13.54%  (p=0.029 n=4+4)
SiteNew/Page_collections-16         58.2ms ± 1%    52.4ms ± 1%   -9.95%  (p=0.029 n=4+4)

name                              old alloc/op   new alloc/op   delta
SiteNew/Bundle_with_image-16        3.81MB ± 0%    2.22MB ± 0%  -41.70%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file-16    3.60MB ± 0%    2.01MB ± 0%  -44.20%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories-16      19.3MB ± 1%    14.1MB ± 0%  -26.91%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs-16            70.7MB ± 0%    69.0MB ± 0%   -2.40%  (p=0.029 n=4+4)
SiteNew/Deep_content_tree-16        37.1MB ± 0%    31.2MB ± 0%  -15.94%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates-16      17.6MB ± 0%    10.6MB ± 0%  -39.92%  (p=0.029 n=4+4)
SiteNew/Page_collections-16         25.9MB ± 0%    21.2MB ± 0%  -17.99%  (p=0.029 n=4+4)

name                              old allocs/op  new allocs/op  delta
SiteNew/Bundle_with_image-16         52.3k ± 0%     26.1k ± 0%  -50.18%  (p=0.029 n=4+4)
SiteNew/Bundle_with_JSON_file-16     52.3k ± 0%     26.1k ± 0%  -50.16%  (p=0.029 n=4+4)
SiteNew/Tags_and_categories-16        336k ± 1%      269k ± 0%  -19.90%  (p=0.029 n=4+4)
SiteNew/Canonify_URLs-16              422k ± 0%      395k ± 0%   -6.43%  (p=0.029 n=4+4)
SiteNew/Deep_content_tree-16          401k ± 0%      313k ± 0%  -21.79%  (p=0.029 n=4+4)
SiteNew/Many_HTML_templates-16        247k ± 0%      143k ± 0%  -42.17%  (p=0.029 n=4+4)
SiteNew/Page_collections-16           282k ± 0%      207k ± 0%  -26.55%  (p=0.029 n=4+4)
```

Fixes #6716
Fixes #6760
Fixes #6768
Fixes #6778
2020-01-22 09:39:49 +01:00
John Zaitseff
158e7ec204 Fix incorrect MIME type from image/jpg to image/jpeg
According to multiple sources, both official ([IANA] and [RFC2046]) and
otherwise (eg, [Mozilla] and [Wikipedia]), the official MIME type for
JPEG images is `image/jpeg`, not `image/jpg`.  Change Hugo to match.

[IANA]: https://www.iana.org/assignments/media-types/media-types.xhtml#image
[RFC2046]: https://tools.ietf.org/html/rfc2046
[Mozilla]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types
[Wikipedia]: https://en.wikipedia.org/wiki/JPEG
2019-12-19 18:02:11 +01:00
Bjørn Erik Pedersen
e625088ef5
Add render template hooks for links and images
This commit also

* revises the change detection for templates used by content files in server mode.
* Adds a Page.RenderString method

Fixes #6545
Fixes #4663
Closes #6043
2019-12-18 11:44:40 +01:00
Bjørn Erik Pedersen
a03c631c42
Rework template handling for function and map lookups
This is a big commit, but it deletes lots of code and simplifies a lot.

* Resolving the template funcs at execution time means we don't have to create template clones per site
* Having a custom map resolver means that we can remove the AST lower case transformation for the special lower case Params map

Not only is the above easier to reason about, it's also faster, especially if you have more than one language, as in the benchmark below:

```
name                          old time/op    new time/op    delta
SiteNew/Deep_content_tree-16    53.7ms ± 0%    48.1ms ± 2%  -10.38%  (p=0.029 n=4+4)

name                          old alloc/op   new alloc/op   delta
SiteNew/Deep_content_tree-16    41.0MB ± 0%    36.8MB ± 0%  -10.26%  (p=0.029 n=4+4)

name                          old allocs/op  new allocs/op  delta
SiteNew/Deep_content_tree-16      481k ± 0%      410k ± 0%  -14.66%  (p=0.029 n=4+4)
```

This should be even better if you also have lots of templates.

Closes #6594
2019-12-12 10:04:35 +01:00
Bjørn Erik Pedersen
0efb00c2a8 tpl/partials: Allow any key type in partialCached
Fixes #6572
2019-12-03 00:13:47 +01:00
Bjørn Erik Pedersen
33d733300a
Deprecate mmark
Fixes #6486
2019-11-25 20:14:20 +01:00
Bjørn Erik Pedersen
d6f7a9e28d resources/images: Make the image cache more robust
Also allow timeout to be set as a duration string, e.g. `30s`.

Fixes #6501
2019-11-25 18:59:06 +01:00
Bjørn Erik Pedersen
a3fe5e5e35
Fix Params case handling in the index, sort and where func
This means that you can now do:

```
{{ range where .Site.Pages "Params.MYPARAM" "foo" }}
```
2019-11-22 18:41:50 +01:00
Luke Francl
70a1aa345b Support Go time format strings in permalinks
In the vein of an ancient TODO about supporting custom date formatting with
strftime, this allows `:`-prefixed Go time format strings in permalink segments.
This allows users to customize date-based permalinks any way they need to.

For example, with a date of 2019-11-09, the permalink `/:06/:1/:2` will render
as `/19/11/9`.

See:

07978e4a49 (diff-0688a3b65c7f5d01aa216f8d9b57fd00R111-R112)
https://discourse.gohugo.io/t/implementing-additional-date-formats-for-permalinks/17860
https://github.com/gohugoio/hugo/pull/6488
2019-11-10 11:00:02 +01:00
Anthony Fok
66fe68ffc9 resources/images: Add exception for new test image
in TestImageOperationsGolden due to "fused multiply and add" (FMA)
instruction on s390x, ppc64* and arm64.

See #6439
2019-10-30 12:22:13 -06:00
Bjørn Erik Pedersen
4b286b9d27 resources/images: Allow to set background fill colour
Closes #6298
2019-10-20 22:06:58 +02:00
Bjørn Erik Pedersen
653e6856ea resources/page: Use binary search in Pages.Prev/Next if possible
This is obviously much faster for lager data sets:

```bash
name                         old time/op    new time/op    delta
SearchPage/ByWeight-100-4       267ns ± 4%     272ns ± 5%     ~     (p=0.457 n=4+4)
SearchPage/ByWeight-5000-4     10.8µs ± 3%     1.2µs ± 2%  -88.99%  (p=0.029 n=4+4)
SearchPage/ByWeight-10000-4    21.1µs ± 1%     1.4µs ±11%  -93.28%  (p=0.029 n=4+4)
```

See #4500
2019-10-13 12:36:17 +02:00
Bjørn Erik Pedersen
f4f566edf4 Make Pages.Prev/Next work like the other Prev/Next methods
Fixes #4500
2019-10-13 12:36:17 +02:00
Anthony Fok
39ed33fceb resources: Fix image test error on s390x, ppc64* and arm64
In TestImageOperationsGolden, tolerate slight floating-point rounding
differences due to the use or non-use of "fused multiply and add" (FMA)
instruction on different architectures.

Special thanks to @disintegration for the solution in goldenEqual();
see https://github.com/disintegration/gift/issues/20

Fixes #6387
2019-10-07 21:52:43 +02:00
Anthony Fok
298092d516 resources: Ensure same dirinfos sort order in TestImageOperationsGolden
Fix filename mismatch errors on Debian auto-building machines
possibly due to different directory order on ext4 vs tmpfs file systems.
2019-10-03 16:48:22 +02:00
J. Ansorg
e5856e61d8 resources: Support output image format in image operations
The image format is defined as the image extension of the known formats,
excluding the dot.
All of 'img.Resize "600x jpeg"', 'img.Resize "600x jpg"',
and 'img.Resize "600x png"' are valid format definitions.
If the target format is defined in the operation definition string,
then the converted image will be stored in this format. Permalinks and
media type are updated correspondingly.
Unknown image extensions in the operation definition have not effect.

See #6298
2019-09-21 16:50:27 +02:00
Bjørn Erik Pedersen
6dec671fb9 Fix cache key transformed resources
Fixes #6348
2019-09-19 12:36:01 +02:00
Bjørn Erik Pedersen
c0d7573677 Fix cache keys for bundled resoures in transform.Unmarshal
Fixes #6327
2019-09-12 20:58:06 +02:00
Bjørn Erik Pedersen
9442937d82 Avoid writing the same processed image to /public twice
Fixes #6307
2019-09-06 11:04:57 +02:00
Bjørn Erik Pedersen
3be2c25351 Fix concat with fingerprint regression
In Hugo 0.58 we optimized the transformers that only adjusted metadata, e.g. the fingerprint.

This depended on the source readers implementing `io.ReadSeeker`.

The reader produced by `concat` did that, but the implementation was buggy.

This commit fixes that.

Fixes #6309
2019-09-05 19:59:49 +02:00
Bjørn Erik Pedersen
bb894ceaf8
Allow slices in the image Filter funcs, not just varargs
[ci skip]

See #6255
2019-09-04 17:24:50 +02:00
Bjørn Erik Pedersen
8624b9fe9e Cache processed images by their source path
Fixes #6269
2019-09-03 18:38:57 +02:00
Bjørn Erik Pedersen
ce47c21a29 resources: Cache Exif data to disk
```bash
name                        old time/op    new time/op    delta
ImageExif/Cold_cache-4         312µs ±28%     355µs ± 7%     ~     (p=0.343 n=4+4)
ImageExif/Cold_cache,_10-4     479µs ± 6%     546µs ± 0%  +13.91%  (p=0.029 n=4+4)
ImageExif/Warm_cache-4         272µs ± 1%      81µs ± 5%  -70.30%  (p=0.029 n=4+4)

name                        old alloc/op   new alloc/op   delta
ImageExif/Cold_cache-4         151kB ± 0%     161kB ± 0%   +6.46%  (p=0.029 n=4+4)
ImageExif/Cold_cache,_10-4     179kB ± 0%     189kB ± 0%   +5.49%  (p=0.029 n=4+4)
ImageExif/Warm_cache-4         151kB ± 0%      13kB ± 0%  -91.52%  (p=0.029 n=4+4)

name                        old allocs/op  new allocs/op  delta
ImageExif/Cold_cache-4         1.03k ± 0%     1.21k ± 0%  +17.78%  (p=0.029 n=4+4)
ImageExif/Cold_cache,_10-4     1.65k ± 0%     1.83k ± 0%  +11.09%  (p=0.029 n=4+4)
ImageExif/Warm_cache-4         1.03k ± 0%     0.28k ± 0%  -72.40%  (p=0.029 n=4+4)
```

Fixes #6291
2019-09-02 16:37:28 +02:00
Bjørn Erik Pedersen
4f5011692a
resources: Make the Exif benchmark filenames distinct 2019-09-01 22:05:39 +02:00
Bjørn Erik Pedersen
3becba7a98
resources: Add Exif benchmark
See #6291
2019-09-01 18:02:49 +02:00
Bjørn Erik Pedersen
20bdc69a47
resources: Remove unused map type 2019-08-31 21:30:13 +02:00
Bjørn Erik Pedersen
28143397d6
Add image.Exif
Note that we will probably need to add some metadata cache for this to scale.

Fixes #4600
2019-08-31 19:04:56 +02:00
Bjørn Erik Pedersen
823f53c861
Add a set of image filters
With this you can do variants of this:

```
{{ $img := resources.Get "images/misc/3-jenny.jpg" }}
{{ $img := $img.Resize "300x" }}
{{ $g1 := $img.Filter images.Grayscale }}
{{ $g2 := $img | images.Filter (images.Saturate 30) (images.GaussianBlur 3) }}
```

Fixes #6255
2019-08-28 15:59:54 +02:00