Commit graph

91 commits

Author SHA1 Message Date
Anthony Fok
3038464ea6
Accept hyphen and plus sign in emoji detection
Fixes #5635
2019-01-29 05:38:36 -07:00
Bjørn Erik Pedersen
be58c7b9c8
tpl/transform: Include options in cache key
See #5555
2018-12-23 21:34:17 +01:00
Bjørn Erik Pedersen
ce06bdb16a
Rename CSV option from comma to delimiter
See #5555
2018-12-23 21:09:09 +01:00
Bjørn Erik Pedersen
a574469797
Add CSV support to transform.Unmarshal
Fixes #5555
2018-12-23 16:33:21 +01:00
Bjørn Erik Pedersen
822dc627a1
tpl/transform: Add transform.Unmarshal func
Fixes #5428
2018-12-23 10:02:42 +01:00
Bjørn Erik Pedersen
9cd54cab20 Move the emoji parsing to pageparser
This avoids double parsing the page content when `enableEmoji=true`.

This commit also adds some general improvements to the parser, making it in general much faster:

```bash
benchmark                     old ns/op     new ns/op     delta
BenchmarkShortcodeLexer-4     90258         101730        +12.71%
BenchmarkParse-4              148940        15037         -89.90%

benchmark                     old allocs     new allocs     delta
BenchmarkShortcodeLexer-4     456            700            +53.51%
BenchmarkParse-4              28             33             +17.86%

benchmark                     old bytes     new bytes     delta
BenchmarkShortcodeLexer-4     69875         81014         +15.94%
BenchmarkParse-4              8128          8304          +2.17%
```

Running some site benchmarks with Emoji support turned on:

```bash
benchmark                                                                                     old ns/op     new ns/op     delta
BenchmarkSiteBuilding/TOML,num_langs=3,num_pages=5000,tags_per_page=5,shortcodes,render-4     924556797     818115620     -11.51%

benchmark                                                                                     old allocs     new allocs     delta
BenchmarkSiteBuilding/TOML,num_langs=3,num_pages=5000,tags_per_page=5,shortcodes,render-4     4112613        4133787        +0.51%

benchmark                                                                                     old bytes     new bytes     delta
BenchmarkSiteBuilding/TOML,num_langs=3,num_pages=5000,tags_per_page=5,shortcodes,render-4     426982864     424363832     -0.61%
```

Fixes #5534
2018-12-20 20:08:01 +01:00
Bjørn Erik Pedersen
a8853f1c5a parser/pageparser: Split the page lexer into some more files
See #5534
2018-12-20 20:08:01 +01:00
Bjørn Erik Pedersen
f2167de834
parser/pageparser: Add a benchmark 2018-12-19 20:08:22 +01:00
Bjørn Erik Pedersen
7829474088
Add /config dir support
This commit adds support for a configuration directory (default `config`). The different pieces in this puzzle are:

* A new `--environment` (or `-e`) flag. This can also be set with the `HUGO_ENVIRONMENT` OS environment variable. The value for `environment` defaults to `production` when running `hugo` and `development` when running `hugo server`. You can set it to any value you want (e.g. `hugo server -e "Sensible Environment"`), but as it is used to load configuration from the file system, the letter case may be important. You can get this value in your templates with `{{ hugo.Environment }}`.
* A new `--configDir` flag (defaults to `config` below your project). This can also be set with `HUGO_CONFIGDIR` OS environment variable.

If the `configDir` exists, the configuration files will be read and merged on top of each other from left to right; the right-most value will win on duplicates.

Given the example tree below:

If `environment` is `production`, the left-most `config.toml` would be the one directly below the project (this can now be omitted if you want), and then `_default/config.toml` and finally `production/config.toml`. And since these will be merged, you can just provide the environment specific configuration setting in you production config, e.g. `enableGitInfo = true`. The order within the directories will be lexical (`config.toml` and then `params.toml`).

```bash
config
├── _default
│   ├── config.toml
│   ├── languages.toml
│   ├── menus
│   │   ├── menus.en.toml
│   │   └── menus.zh.toml
│   └── params.toml
├── development
│   └── params.toml
└── production
    ├── config.toml
    └── params.toml
```

Some configuration maps support the language code in the filename (e.g. `menus.en.toml`): `menus` (`menu` also works) and `params`.

Also note that the only folders with "a meaning" in the above listing is the top level directories below `config`. The `menus` sub folder is just added for better organization.

We use `TOML` in the example above, but Hugo also supports `JSON` and `YAML` as configuration formats. These can be mixed.

Fixes #5422
2018-12-11 13:08:36 +01:00
Bjørn Erik Pedersen
7540a62834
parser/pageparser: Fix handling of commented out front matter
When the page parser was rewritten in 0.51, this was interpreted literally, but commented out front matter is used in the wild to "hide it from GitHub", e.g:

```
<!--
+++
title = "hello"
+++
-->
```

Fixes #5478
2018-11-28 10:28:50 +01:00
Bjørn Erik Pedersen
bc337e6ab5 Add inline shortcode support
An inline shortcode's name must end with `.inline`, all lowercase.

E.g.:

```bash
{{< time.inline >}}{{ now }}{{< /time.inline >}}
```

The above will print the current date and time.

Note that an inline shortcode's inner content is parsed and executed as a Go text template with the same context as a regular shortcode template.

This means that the current page can be accessed via `.Page.Title` etc. This also means that there are no concept of "nested inline shortcodes".

The same inline shortcode can be reused later in the same content file, with different params if needed, using the self-closing syntax:

```
{{< time.inline />}}
```

Fixes #4011
2018-11-27 16:14:09 +01:00
Bjørn Erik Pedersen
94ab125b27 parser/pageparser: Fix when only shortcode and then summary
Fixes #5464
2018-11-24 18:23:23 +01:00
Bjørn Erik Pedersen
782dd15858
More spelling corrections 2018-11-13 18:28:40 +01:00
Bjørn Erik Pedersen
d16a7a33ff Fix shortcode directly following a shortcode delimiter
Fixes #5402
2018-11-05 14:14:28 +01:00
Bjørn Erik Pedersen
b2a676f5f0 hugolib: Fix broken manual summary handling
Fixes #5381
2018-10-31 23:14:37 +01:00
Bjørn Erik Pedersen
6636cf1bea
Resolve error handling/parser related TODOs
See #5324
2018-10-23 19:41:22 +02:00
Bjørn Erik Pedersen
f669ef6bec
herrors: Improve handling of JSON errors
`*json.UnmarshalTypeError` and `*json.SyntaxError` has a byte `Offset`, so use that.

This commit also reworks/simplifies the errror line matching logic. This also makes the file reading unbuffered, but that should be fine in this error case.

See #5324
2018-10-23 14:35:43 +02:00
Bjørn Erik Pedersen
d1661b823a
hugolib: Continue the file context/line number errors work
See #5324
2018-10-22 20:46:14 +02:00
Bjørn Erik Pedersen
7930d2132a
hugolib: Remove the now superflous Source struct
See #5324
2018-10-22 20:46:14 +02:00
Bjørn Erik Pedersen
eb038cfa0a
Convert the rest to new page parser code paths
And remove some now unused code.

See #5324
2018-10-22 20:46:14 +02:00
Bjørn Erik Pedersen
129c27ee6e
parser/metadecoders: Consolidate the metadata decoders
See #5324
2018-10-22 20:46:13 +02:00
Bjørn Erik Pedersen
44da60d869
hugolib: Redo the summary delimiter logic
Now that we have a proper page parse tree, this can be greatly simplified.

See #5324
2018-10-22 20:46:13 +02:00
Bjørn Erik Pedersen
1e3e34002d
hugolib: Integrate new page parser
See #5324
2018-10-22 20:46:13 +02:00
Bjørn Erik Pedersen
1b7ecfc2e1
hugolib: Use []byte in shortcode parsing
See #5324
2018-10-22 19:57:44 +02:00
Bjørn Erik Pedersen
27f5a906a2
parser/pageparser: Use []byte in page lexer
See #5324
2018-10-22 19:57:44 +02:00
Bjørn Erik Pedersen
2fdc4a24d5
parser/pageparser: Add front matter etc. support
See #5324
2018-10-22 19:57:43 +02:00
Bjørn Erik Pedersen
f6863e1ef7
parser/pageparser: File renames and splitting
See #5324
2018-10-22 19:57:43 +02:00
Bjørn Erik Pedersen
d6c16afde0
Move the shortcode parser to the new pageparser package
See #5324
2018-10-22 19:57:43 +02:00
Bjørn Erik Pedersen
dea71670c0
Add Hugo Piper with SCSS support and much more
Before this commit, you would have to use page bundles to do image processing etc. in Hugo.

This commit adds

* A new `/assets` top-level project or theme dir (configurable via `assetDir`)
* A new template func, `resources.Get` which can be used to "get a resource" that can be further processed.

This means that you can now do this in your templates (or shortcodes):

```bash
{{ $sunset := (resources.Get "images/sunset.jpg").Fill "300x200" }}
```

This also adds a new `extended` build tag that enables powerful SCSS/SASS support with source maps. To compile this from source, you will also need a C compiler installed:

```
HUGO_BUILD_TAGS=extended mage install
```

Note that you can use output of the SCSS processing later in a non-SCSSS-enabled Hugo.

The `SCSS` processor is a _Resource transformation step_ and it can be chained with the many others in a pipeline:

```bash
{{ $css := resources.Get "styles.scss" | resources.ToCSS | resources.PostCSS | resources.Minify | resources.Fingerprint }}
<link rel="stylesheet" href="{{ $styles.RelPermalink }}" integrity="{{ $styles.Data.Digest }}" media="screen">
```

The transformation funcs above have aliases, so it can be shortened to:

```bash
{{ $css := resources.Get "styles.scss" | toCSS | postCSS | minify | fingerprint }}
<link rel="stylesheet" href="{{ $styles.RelPermalink }}" integrity="{{ $styles.Data.Digest }}" media="screen">
```

A quick tip would be to avoid the fingerprinting part, and possibly also the not-superfast `postCSS` when you're doing development, as it allows Hugo to be smarter about the rebuilding.

Documentation will follow, but have a look at the demo repo in https://github.com/bep/hugo-sass-test

New functions to create `Resource` objects:

* `resources.Get` (see above)
* `resources.FromString`: Create a Resource from a string.

New `Resource` transformation funcs:

* `resources.ToCSS`: Compile `SCSS` or `SASS` into `CSS`.
* `resources.PostCSS`: Process your CSS with PostCSS. Config file support (project or theme or passed as an option).
* `resources.Minify`: Currently supports `css`, `js`, `json`, `html`, `svg`, `xml`.
* `resources.Fingerprint`: Creates a fingerprinted version of the given Resource with Subresource Integrity..
* `resources.Concat`: Concatenates a list of Resource objects. Think of this as a poor man's bundler.
* `resources.ExecuteAsTemplate`: Parses and executes the given Resource and data context (e.g. .Site) as a Go template.

Fixes #4381
Fixes #4903
Fixes #4858
2018-07-06 11:46:12 +02:00
Bjørn Erik Pedersen
159bed34c3
parser: Add some context to front matter parse error
Fixes #4638
2018-04-17 10:20:38 +02:00
Bjørn Erik Pedersen
0816a97a46
parser: Add WARNING for integer YAML keys
```bash
benchmark                                               old ns/op     new ns/op     delta
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4     3053          2015          -34.00%
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        5.23          5.18          -0.96%
BenchmarkStringifyMapKeysIntegers-4                     2320          5177          +123.15%

benchmark                                               old allocs     new allocs     delta
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4     6              6              +0.00%
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        0              0              +0.00%
BenchmarkStringifyMapKeysIntegers-4                     6              14             +133.33%

benchmark                                               old bytes     new bytes     delta
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4     1008          1008          +0.00%
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        0             0             +0.00%
BenchmarkStringifyMapKeysIntegers-4                     1008          1776          +76.19%
```
Closes #4393
2018-02-12 19:16:12 +01:00
Bjørn Erik Pedersen
10a917dfdc
parser: Tune stringifyMapKeys
```bash
benchmark                                               old ns/op     new ns/op     delta
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4     3269          3053          -6.61%
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        4.79          5.23          +9.19%
BenchmarkStringifyMapKeysIntegers-4                     2707          2320          -14.30%

benchmark                                               old allocs     new allocs     delta
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4     16             6              -62.50%
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        0              0              +0.00%
BenchmarkStringifyMapKeysIntegers-4                     16             6              -62.50%

benchmark                                               old bytes     new bytes     delta
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4     1080          1008          -6.67%
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        0             0             +0.00%
BenchmarkStringifyMapKeysIntegers-4                     1080          1008          -6.67%
```
2018-02-12 19:14:02 +01:00
Bjørn Erik Pedersen
d4beef0d2b
parser: Rename stringifyYAMLMapKeys to stringifyMapKeys 2018-02-12 18:17:25 +01:00
Bjørn Erik Pedersen
51213e0be1
parser: Add benchmarks for stringifyYAMLMapKeys
```bash
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4       500000              3269 ns/op            1080 B/op         16 allocs/op
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        300000000                4.79 ns/op            0 B/op          0 allocs/op
BenchmarkStringifyMapKeysIntegers-4                       500000              2707 ns/op            1080 B/op         16 allocs/op
```
2018-02-12 18:16:48 +01:00
Vas Sudanagunta
1fa2417777 Add support for YAML array data files
* Unmarshaled YAML arrays indistinguishable from JSON arrays.
* Fixes #3890
2018-02-12 17:14:40 +01:00
Dawid Gaweł
16a5c74519 parser: Fix YAML maps key type
Recurse through result of yaml package parsing and change all
maps from map[interface{}]interface{} to map[string]interface{}
making them jsonable and sortable.

Fixes #2441, #4083
2018-02-09 10:27:17 +01:00
Vas Sudanagunta
4402c07775 Fix JSON array-based data file handling regression
This bug was introduced in Hugo 0.35.

Fixes #4361
2018-02-02 09:14:37 +01:00
Vas Sudanagunta
91bb774ae4 Support pages without front matter
* Page without front matter now treated same as a page with empty front matter.
* Test cases added to cover this and repro issue #4320.
* Type safety of front matter code improved.

Fixes #4320
2018-01-26 09:17:27 +01:00
Bjørn Erik Pedersen
7f82b41a24 parser: Final (!) fix for issue with escaped JSON front matter
Fixes #3682
2017-07-08 18:43:36 +02:00
Bjørn Erik Pedersen
84db6c74a0 parser: Fix issue with escaped JSON front matter
Fixes #3682
2017-07-08 10:56:56 +02:00
Bjørn Erik Pedersen
e10e51a008 parser: Fix handling of JSON front matter with escaped quotes
Fixes #3661
2017-07-03 09:01:54 +02:00
Bjørn Erik Pedersen
3183b9a29d parser: Fix handling of quoted brackets in JSON front matter
Also remove a broken JSON test.

Fixes #3511
2017-06-19 16:45:52 +02:00
Bjørn Erik Pedersen
422057f607 create: Use archetype template as-is as a Go template
This commit removes the fragile front matter decoding, and takes the provided archetype file as-is and processes it as a template.

This also means that we no longer will attempt to fill in default values for `title` and `date`.

The upside is that it is now easy to create these values in a dynamic way:

```toml
+++
title = {{ .BaseFileName | title }}
date = {{ .Date }}
draft = true
+++
```

You can currently use all of Hugo's template funcs, but the data context is currently very shallow:

* `.Type` gives the archetype kind provided
* `.Name` gives the target file name without extension.
* `.Path` gives the target file name
* `.Date` gives the current time as RFC3339 formatted string

The above  will probably be extended in #1629.

Fixes #452
Updates #1629
2017-06-18 19:06:28 +02:00
Bjørn Erik Pedersen
2c2ce33a39 parser: Add horizontal YAML tags to benchmark 2017-06-03 12:35:28 +02:00
Bjørn Erik Pedersen
0907a5c1c2 all: Temporarily revert to BurntSushi for TOML front matter handling
We still have go-toml as a transitive dependency, and it is the way to go eventually, but we care about speed, so let us wait that one out.

Note that the issue this fixes is about taxonomies, but I guess this is a general issue for sites with many pages that uses TOML as front matter.

```
benchmark                              old ns/op     new ns/op     delta
BenchmarkFrontmatterTags/TOML:1-4      23206         8543          -63.19%
BenchmarkFrontmatterTags/TOML:11-4     80117         18495         -76.92%
BenchmarkFrontmatterTags/TOML:21-4     140676        28727         -79.58%

benchmark                              old allocs     new allocs     delta
BenchmarkFrontmatterTags/TOML:1-4      173            60             -65.32%
BenchmarkFrontmatterTags/TOML:11-4     625            138            -77.92%
BenchmarkFrontmatterTags/TOML:21-4     1106           210            -81.01%

benchmark                              old bytes     new bytes     delta
BenchmarkFrontmatterTags/TOML:1-4      9231          2912          -68.45%
BenchmarkFrontmatterTags/TOML:11-4     19808         5184          -73.83%
BenchmarkFrontmatterTags/TOML:21-4     31200         7536          -75.85%
```

See #3541
Updates #3464
2017-06-03 09:22:57 +02:00
Bjørn Erik Pedersen
3d9c4f513b parser: Add BenchmarkFrontmatterTags
The list handling is surprisingly expensive:

```
▶ go test -run="NONE" -bench="BenchmarkFrontmatterTags" -test.benchmem=true ./parser | prettybench
PASS
benchmark                               iter         time/iter   bytes alloc           allocs
---------                               ----         ---------   -----------           ------
BenchmarkFrontmatterTags/JSON:1-4    1000000     2039.00 ns/op      912 B/op     20 allocs/op
BenchmarkFrontmatterTags/JSON:11-4    300000     5202.00 ns/op     1640 B/op     44 allocs/op
BenchmarkFrontmatterTags/JSON:21-4    200000     7993.00 ns/op     2392 B/op     65 allocs/op
BenchmarkFrontmatterTags/YAML:1-4     200000     9359.00 ns/op     5928 B/op     66 allocs/op
BenchmarkFrontmatterTags/YAML:11-4    100000    21218.00 ns/op     8408 B/op    140 allocs/op
BenchmarkFrontmatterTags/YAML:21-4     50000    32852.00 ns/op    10920 B/op    211 allocs/op
BenchmarkFrontmatterTags/TOML:1-4     100000    21505.00 ns/op     9231 B/op    173 allocs/op
BenchmarkFrontmatterTags/TOML:11-4     20000    82919.00 ns/op    19808 B/op    625 allocs/op
BenchmarkFrontmatterTags/TOML:21-4     10000   141847.00 ns/op    31200 B/op   1106 allocs/op
ok      github.com/spf13/hugo/parser    17.890s
```

See #3464
2017-06-03 08:53:56 +02:00
Albert Nigmatzianov
d3cd68e656 parser: Improve TOML frontmatter parser performance
Difference between toml.Load(string(datum)) and
toml.LoadReader(bytes.NewReader(datum)):
benchmark           old ns/op     new ns/op     delta
BenchmarkLoad-4     82068         78489         -4.36%

benchmark           old allocs     new allocs     delta
BenchmarkLoad-4     494            493            -0.20%

benchmark           old bytes     new bytes     delta
BenchmarkLoad-4     17009         16913         -0.56%
2017-03-20 09:02:35 +01:00
Bjørn Erik Pedersen
d8923eb676 Update to new go-toml API
Closes #3142
2017-03-15 08:04:52 +01:00
crasm
ede452d34e parser: Accept JSON frontmatter without leading "{\n"
Accept JSON frontmatter without leading "{\n" so that one line
frontmatters such as `{"param":"paramvalue"}` no longer silently render
empty html.
2017-03-13 17:19:03 +01:00
Cameron Moore
f039e3be9e parser: Refactor frontmatter parser and add tests
Lots of cleanups here:

- Refactor InterfaceToConfig and InterfaceToFrontMatter to use io.Writer.
- Simplify InterfaceToFrontMatter by wrapping InterfaceToConfig.
- Export FrontmatterType since we return it in DetectFrontMatter.
- Refactor removeTOMLIdentifier to avoid blindly replacing "+++".
- Update HandleJSONMetaData to return an empty map on nil input.
- Updates vendored goorgeous package and test for org-mode frontmatter.
- Add tests and godoc comments.

Coverage for parser package increased from 45.2% to 85.2%.
2017-03-11 17:52:25 +01:00