Implement XML data support

Example:

```
{{ with resources.Get "https://example.com/rss.xml" | transform.Unmarshal }}
    {{ range .channel.item }}
        <strong>{{ .title | plainify | htmlUnescape }}</strong><br />
        <p>{{ .description | plainify | htmlUnescape }}</p>
        {{ $link := .link | plainify | htmlUnescape }}
        <a href="{{ $link }}">{{ $link }}</a><br />
        <hr>
    {{ end }}
{{ end }}
```

Closes #4470
This commit is contained in:
Paul van Brouwershaven 2021-12-02 17:30:36 +01:00 committed by GitHub
parent 58adbeef88
commit 0eaaa8fee3
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
12 changed files with 167 additions and 12 deletions

View file

@ -1,6 +1,6 @@
--- ---
title: "transform.Unmarshal" title: "transform.Unmarshal"
description: "`transform.Unmarshal` (alias `unmarshal`) parses the input and converts it into a map or an array. Supported formats are JSON, TOML, YAML and CSV." description: "`transform.Unmarshal` (alias `unmarshal`) parses the input and converts it into a map or an array. Supported formats are JSON, TOML, YAML, XML and CSV."
date: 2018-12-23 date: 2018-12-23
categories: [functions] categories: [functions]
menu: menu:
@ -45,3 +45,32 @@ Example:
```go-html-template ```go-html-template
{{ $csv := "a;b;c" | transform.Unmarshal (dict "delimiter" ";") }} {{ $csv := "a;b;c" | transform.Unmarshal (dict "delimiter" ";") }}
``` ```
## XML data
As a convenience, Hugo allows you to access XML data in the same way that you access JSON, TOML, and YAML: you do not need to specify the root node when accessing the data.
To get the contents of `<title>` in the document below, you use `{{ .message.title }}`:
```
<root>
<message>
<title>Hugo rocks!</title>
<description>Thanks for using Hugo</description>
</message>
</root>
```
The following example lists the items of an RSS feed:
```
{{ with resources.Get "https://example.com/rss.xml" | transform.Unmarshal }}
{{ range .channel.item }}
<strong>{{ .title | plainify | htmlUnescape }}</strong><br />
<p>{{ .description | plainify | htmlUnescape }}</p>
{{ $link := .link | plainify | htmlUnescape }}
<a href="{{ $link }}">{{ $link }}</a><br />
<hr>
{{ end }}
{{ end }}
```

View file

@ -6,7 +6,7 @@ date: 2017-02-01
publishdate: 2017-02-01 publishdate: 2017-02-01
lastmod: 2017-03-12 lastmod: 2017-03-12
categories: [templates] categories: [templates]
keywords: [data,dynamic,csv,json,toml,yaml] keywords: [data,dynamic,csv,json,toml,yaml,xml]
menu: menu:
docs: docs:
parent: "templates" parent: "templates"
@ -20,7 +20,7 @@ toc: true
<!-- begin data files --> <!-- begin data files -->
Hugo supports loading data from YAML, JSON, and TOML files located in the `data` directory in the root of your Hugo project. Hugo supports loading data from YAML, JSON, XML, and TOML files located in the `data` directory in the root of your Hugo project.
{{< youtube FyPgSuwIMWQ >}} {{< youtube FyPgSuwIMWQ >}}
@ -28,7 +28,7 @@ Hugo supports loading data from YAML, JSON, and TOML files located in the `data`
The `data` folder is where you can store additional data for Hugo to use when generating your site. Data files aren't used to generate standalone pages; rather, they're meant to be supplemental to content files. This feature can extend the content in case your front matter fields grow out of control. Or perhaps you want to show a larger dataset in a template (see example below). In both cases, it's a good idea to outsource the data in their own files. The `data` folder is where you can store additional data for Hugo to use when generating your site. Data files aren't used to generate standalone pages; rather, they're meant to be supplemental to content files. This feature can extend the content in case your front matter fields grow out of control. Or perhaps you want to show a larger dataset in a template (see example below). In both cases, it's a good idea to outsource the data in their own files.
These files must be YAML, JSON, or TOML files (using the `.yml`, `.yaml`, `.json`, or `.toml` extension). The data will be accessible as a `map` in the `.Site.Data` variable. These files must be YAML, JSON, XML, or TOML files (using the `.yml`, `.yaml`, `.json`, `.xml`, or `.toml` extension). The data will be accessible as a `map` in the `.Site.Data` variable.
## Data Files in Themes ## Data Files in Themes
@ -95,7 +95,7 @@ Discover a new favorite bass player? Just add another `.toml` file in the same d
## Example: Accessing Named Values in a Data File ## Example: Accessing Named Values in a Data File
Assume you have the following data structure in your `User0123.[yml|toml|json]` data file located directly in `data/`: Assume you have the following data structure in your `User0123.[yml|toml|xml|json]` data file located directly in `data/`:
{{< code-toggle file="User0123" >}} {{< code-toggle file="User0123" >}}
Name: User0123 Name: User0123
@ -232,6 +232,7 @@ If you change any local file and the LiveReload is triggered, Hugo will read the
* [YAML Spec][yaml] * [YAML Spec][yaml]
* [JSON Spec][json] * [JSON Spec][json]
* [CSV Spec][csv] * [CSV Spec][csv]
* [XML Spec][xml]
[config]: /getting-started/configuration/ [config]: /getting-started/configuration/
[csv]: https://tools.ietf.org/html/rfc4180 [csv]: https://tools.ietf.org/html/rfc4180
@ -247,3 +248,4 @@ If you change any local file and the LiveReload is triggered, Hugo will read the
[variadic]: https://en.wikipedia.org/wiki/Variadic_function [variadic]: https://en.wikipedia.org/wiki/Variadic_function
[vars]: /variables/ [vars]: /variables/
[yaml]: https://yaml.org/spec/ [yaml]: https://yaml.org/spec/
[xml]: https://www.w3.org/XML/

1
go.mod
View file

@ -13,6 +13,7 @@ require (
github.com/bep/golibsass v1.0.0 github.com/bep/golibsass v1.0.0
github.com/bep/gowebp v0.1.0 github.com/bep/gowebp v0.1.0
github.com/bep/tmc v0.5.1 github.com/bep/tmc v0.5.1
github.com/clbanning/mxj/v2 v2.5.5
github.com/cli/safeexec v1.0.0 github.com/cli/safeexec v1.0.0
github.com/disintegration/gift v1.2.1 github.com/disintegration/gift v1.2.1
github.com/dustin/go-humanize v1.0.0 github.com/dustin/go-humanize v1.0.0

2
go.sum
View file

@ -144,6 +144,8 @@ github.com/cheekybits/is v0.0.0-20150225183255-68e9c0620927/go.mod h1:h/aW8ynjgk
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/clbanning/mxj/v2 v2.5.5 h1:oT81vUeEiQQ/DcHbzSytRngP6Ky9O+L+0Bw0zSJag9E=
github.com/clbanning/mxj/v2 v2.5.5/go.mod h1:hNiWqW14h+kc+MdF9C6/YoRfjEJoR3ou6tn/Qo+ve2s=
github.com/cli/safeexec v1.0.0 h1:0VngyaIyqACHdcMNWfo6+KdUYnqEr2Sg+bSP1pdF+dI= github.com/cli/safeexec v1.0.0 h1:0VngyaIyqACHdcMNWfo6+KdUYnqEr2Sg+bSP1pdF+dI=
github.com/cli/safeexec v1.0.0/go.mod h1:Z/D4tTN8Vs5gXYHDCbaM1S/anmEDnJb1iW0+EJ5zx3Q= github.com/cli/safeexec v1.0.0/go.mod h1:Z/D4tTN8Vs5gXYHDCbaM1S/anmEDnJb1iW0+EJ5zx3Q=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=

View file

@ -872,16 +872,19 @@ Publish 2: {{ $cssPublish2.Permalink }}
{{ $toml := "slogan = \"Hugo Rocks!\"" | resources.FromString "slogan.toml" | transform.Unmarshal }} {{ $toml := "slogan = \"Hugo Rocks!\"" | resources.FromString "slogan.toml" | transform.Unmarshal }}
{{ $csv1 := "\"Hugo Rocks\",\"Hugo is Fast!\"" | resources.FromString "slogans.csv" | transform.Unmarshal }} {{ $csv1 := "\"Hugo Rocks\",\"Hugo is Fast!\"" | resources.FromString "slogans.csv" | transform.Unmarshal }}
{{ $csv2 := "a;b;c" | transform.Unmarshal (dict "delimiter" ";") }} {{ $csv2 := "a;b;c" | transform.Unmarshal (dict "delimiter" ";") }}
{{ $xml := "<?xml version=\"1.0\" encoding=\"UTF-8\"?><note><to>You</to><from>Me</from><heading>Reminder</heading><body>Do not forget XML</body></note>" | transform.Unmarshal }}
Slogan: {{ $toml.slogan }} Slogan: {{ $toml.slogan }}
CSV1: {{ $csv1 }} {{ len (index $csv1 0) }} CSV1: {{ $csv1 }} {{ len (index $csv1 0) }}
CSV2: {{ $csv2 }} CSV2: {{ $csv2 }}
XML: {{ $xml.body }}
`) `)
}, func(b *sitesBuilder) { }, func(b *sitesBuilder) {
b.AssertFileContent("public/index.html", b.AssertFileContent("public/index.html",
`Slogan: Hugo Rocks!`, `Slogan: Hugo Rocks!`,
`[[Hugo Rocks Hugo is Fast!]] 2`, `[[Hugo Rocks Hugo is Fast!]] 2`,
`CSV2: [[a b c]]`, `CSV2: [[a b c]]`,
`XML: Do not forget XML`,
) )
}}, }},
{"resources.Get", func() bool { return true }, func(b *sitesBuilder) { {"resources.Get", func() bool { return true }, func(b *sitesBuilder) {

View file

@ -23,6 +23,8 @@ import (
toml "github.com/pelletier/go-toml/v2" toml "github.com/pelletier/go-toml/v2"
yaml "gopkg.in/yaml.v2" yaml "gopkg.in/yaml.v2"
xml "github.com/clbanning/mxj/v2"
) )
const ( const (
@ -62,7 +64,14 @@ func InterfaceToConfig(in interface{}, format metadecoders.Format, w io.Writer)
_, err = w.Write([]byte{'\n'}) _, err = w.Write([]byte{'\n'})
return err return err
case metadecoders.XML:
b, err := xml.AnyXmlIndent(in, "", "\t", "root")
if err != nil {
return err
}
_, err = w.Write(b)
return err
default: default:
return errors.New("unsupported Format provided") return errors.New("unsupported Format provided")
} }

View file

@ -24,6 +24,7 @@ import (
"github.com/gohugoio/hugo/common/herrors" "github.com/gohugoio/hugo/common/herrors"
"github.com/niklasfasching/go-org/org" "github.com/niklasfasching/go-org/org"
xml "github.com/clbanning/mxj/v2"
toml "github.com/pelletier/go-toml/v2" toml "github.com/pelletier/go-toml/v2"
"github.com/pkg/errors" "github.com/pkg/errors"
"github.com/spf13/afero" "github.com/spf13/afero"
@ -135,6 +136,25 @@ func (d Decoder) UnmarshalTo(data []byte, f Format, v interface{}) error {
err = d.unmarshalORG(data, v) err = d.unmarshalORG(data, v)
case JSON: case JSON:
err = json.Unmarshal(data, v) err = json.Unmarshal(data, v)
case XML:
var xmlRoot xml.Map
xmlRoot, err = xml.NewMapXml(data)
var xmlValue map[string]interface{}
if err == nil {
xmlRootName, err := xmlRoot.Root()
if err != nil {
return toFileError(f, errors.Wrap(err, "failed to unmarshal XML"))
}
xmlValue = xmlRoot[xmlRootName].(map[string]interface{})
}
switch v := v.(type) {
case *map[string]interface{}:
*v = xmlValue
case *interface{}:
*v = xmlValue
}
case TOML: case TOML:
err = toml.Unmarshal(data, v) err = toml.Unmarshal(data, v)
case YAML: case YAML:

View file

@ -20,6 +20,59 @@ import (
qt "github.com/frankban/quicktest" qt "github.com/frankban/quicktest"
) )
func TestUnmarshalXML(t *testing.T) {
c := qt.New(t)
xmlDoc := `<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0"
xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Example feed</title>
<link>https://example.com/</link>
<description>Example feed</description>
<generator>Hugo -- gohugo.io</generator>
<language>en-us</language>
<copyright>Example</copyright>
<lastBuildDate>Fri, 08 Jan 2021 14:44:10 +0000</lastBuildDate>
<atom:link href="https://example.com/feed.xml" rel="self" type="application/rss+xml"/>
<item>
<title>Example title</title>
<link>https://example.com/2021/11/30/example-title/</link>
<pubDate>Tue, 30 Nov 2021 15:00:00 +0000</pubDate>
<guid>https://example.com/2021/11/30/example-title/</guid>
<description>Example description</description>
</item>
</channel>
</rss>`
expect := map[string]interface{}{
"-atom": "http://www.w3.org/2005/Atom", "-version": "2.0",
"channel": map[string]interface{}{
"copyright": "Example",
"description": "Example feed",
"generator": "Hugo -- gohugo.io",
"item": map[string]interface{}{
"description": "Example description",
"guid": "https://example.com/2021/11/30/example-title/",
"link": "https://example.com/2021/11/30/example-title/",
"pubDate": "Tue, 30 Nov 2021 15:00:00 +0000",
"title": "Example title"},
"language": "en-us",
"lastBuildDate": "Fri, 08 Jan 2021 14:44:10 +0000",
"link": []interface{}{"https://example.com/", map[string]interface{}{
"-href": "https://example.com/feed.xml",
"-rel": "self",
"-type": "application/rss+xml"}},
"title": "Example feed",
}}
d := Default
m, err := d.Unmarshal([]byte(xmlDoc), XML)
c.Assert(err, qt.IsNil)
c.Assert(m, qt.DeepEquals, expect)
}
func TestUnmarshalToMap(t *testing.T) { func TestUnmarshalToMap(t *testing.T) {
c := qt.New(t) c := qt.New(t)
@ -38,6 +91,7 @@ func TestUnmarshalToMap(t *testing.T) {
{"a: Easy!\nb:\n c: 2\n d: [3, 4]", YAML, map[string]interface{}{"a": "Easy!", "b": map[string]interface{}{"c": 2, "d": []interface{}{3, 4}}}}, {"a: Easy!\nb:\n c: 2\n d: [3, 4]", YAML, map[string]interface{}{"a": "Easy!", "b": map[string]interface{}{"c": 2, "d": []interface{}{3, 4}}}},
{"a:\n true: 1\n false: 2", YAML, map[string]interface{}{"a": map[string]interface{}{"true": 1, "false": 2}}}, {"a:\n true: 1\n false: 2", YAML, map[string]interface{}{"a": map[string]interface{}{"true": 1, "false": 2}}},
{`{ "a": "b" }`, JSON, expect}, {`{ "a": "b" }`, JSON, expect},
{`<root><a>b</a></root>`, XML, expect},
{`#+a: b`, ORG, expect}, {`#+a: b`, ORG, expect},
// errors // errors
{`a = b`, TOML, false}, {`a = b`, TOML, false},
@ -72,6 +126,7 @@ func TestUnmarshalToInterface(t *testing.T) {
{`#+DATE: <2020-06-26 Fri>`, ORG, map[string]interface{}{"date": "2020-06-26"}}, {`#+DATE: <2020-06-26 Fri>`, ORG, map[string]interface{}{"date": "2020-06-26"}},
{`a = "b"`, TOML, expect}, {`a = "b"`, TOML, expect},
{`a: "b"`, YAML, expect}, {`a: "b"`, YAML, expect},
{`<root><a>b</a></root>`, XML, expect},
{`a,b,c`, CSV, [][]string{{"a", "b", "c"}}}, {`a,b,c`, CSV, [][]string{{"a", "b", "c"}}},
{"a: Easy!\nb:\n c: 2\n d: [3, 4]", YAML, map[string]interface{}{"a": "Easy!", "b": map[string]interface{}{"c": 2, "d": []interface{}{3, 4}}}}, {"a: Easy!\nb:\n c: 2\n d: [3, 4]", YAML, map[string]interface{}{"a": "Easy!", "b": map[string]interface{}{"c": 2, "d": []interface{}{3, 4}}}},
// errors // errors

View file

@ -30,6 +30,7 @@ const (
TOML Format = "toml" TOML Format = "toml"
YAML Format = "yaml" YAML Format = "yaml"
CSV Format = "csv" CSV Format = "csv"
XML Format = "xml"
) )
// FormatFromString turns formatStr, typically a file extension without any ".", // FormatFromString turns formatStr, typically a file extension without any ".",
@ -51,6 +52,8 @@ func FormatFromString(formatStr string) Format {
return ORG return ORG
case "csv": case "csv":
return CSV return CSV
case "xml":
return XML
} }
return "" return ""
@ -68,27 +71,32 @@ func FormatFromMediaType(m media.Type) Format {
return "" return ""
} }
// FormatFromContentString tries to detect the format (JSON, YAML or TOML) // FormatFromContentString tries to detect the format (JSON, YAML, TOML or XML)
// in the given string. // in the given string.
// It return an empty string if no format could be detected. // It return an empty string if no format could be detected.
func (d Decoder) FormatFromContentString(data string) Format { func (d Decoder) FormatFromContentString(data string) Format {
csvIdx := strings.IndexRune(data, d.Delimiter) csvIdx := strings.IndexRune(data, d.Delimiter)
jsonIdx := strings.Index(data, "{") jsonIdx := strings.Index(data, "{")
yamlIdx := strings.Index(data, ":") yamlIdx := strings.Index(data, ":")
xmlIdx := strings.Index(data, "<")
tomlIdx := strings.Index(data, "=") tomlIdx := strings.Index(data, "=")
if isLowerIndexThan(csvIdx, jsonIdx, yamlIdx, tomlIdx) { if isLowerIndexThan(csvIdx, jsonIdx, yamlIdx, xmlIdx, tomlIdx) {
return CSV return CSV
} }
if isLowerIndexThan(jsonIdx, yamlIdx, tomlIdx) { if isLowerIndexThan(jsonIdx, yamlIdx, xmlIdx, tomlIdx) {
return JSON return JSON
} }
if isLowerIndexThan(yamlIdx, tomlIdx) { if isLowerIndexThan(yamlIdx, xmlIdx, tomlIdx) {
return YAML return YAML
} }
if isLowerIndexThan(xmlIdx, tomlIdx) {
return XML
}
if tomlIdx != -1 { if tomlIdx != -1 {
return TOML return TOML
} }

View file

@ -30,6 +30,7 @@ func TestFormatFromString(t *testing.T) {
{"json", JSON}, {"json", JSON},
{"yaml", YAML}, {"yaml", YAML},
{"yml", YAML}, {"yml", YAML},
{"xml", XML},
{"toml", TOML}, {"toml", TOML},
{"config.toml", TOML}, {"config.toml", TOML},
{"tOMl", TOML}, {"tOMl", TOML},
@ -48,6 +49,7 @@ func TestFormatFromMediaType(t *testing.T) {
}{ }{
{media.JSONType, JSON}, {media.JSONType, JSON},
{media.YAMLType, YAML}, {media.YAMLType, YAML},
{media.XMLType, XML},
{media.TOMLType, TOML}, {media.TOMLType, TOML},
{media.CalendarType, ""}, {media.CalendarType, ""},
} { } {
@ -70,6 +72,7 @@ func TestFormatFromContentString(t *testing.T) {
{`foo:"bar"`, YAML}, {`foo:"bar"`, YAML},
{`{ "foo": "bar"`, JSON}, {`{ "foo": "bar"`, JSON},
{`a,b,c"`, CSV}, {`a,b,c"`, CSV},
{`<foo>bar</foo>"`, XML},
{`asdfasdf`, Format("")}, {`asdfasdf`, Format("")},
{``, Format("")}, {``, Format("")},
} { } {

View file

@ -81,6 +81,25 @@ title: Test Metadata
], ],
"title": "Test Metadata" "title": "Test Metadata"
} }
`
xmlExample := `<root>
<resources>
<params>
<byline>picasso</byline>
</params>
<src>**image-4.png</src>
<title>The Fourth Image!</title>
</resources>
<resources>
<name>my-cool-image-:counter</name>
<params>
<byline>bep</byline>
</params>
<src>**.png</src>
<title>TOML: The Image #:counter</title>
</resources>
<title>Test Metadata</title>
</root>
` `
variants := []struct { variants := []struct {
@ -93,6 +112,7 @@ title: Test Metadata
{"TOML", tomlExample}, {"TOML", tomlExample},
{"Toml", tomlExample}, {"Toml", tomlExample},
{" TOML ", tomlExample}, {" TOML ", tomlExample},
{"XML", xmlExample},
} }
for _, v1 := range variants { for _, v1 := range variants {

View file

@ -111,6 +111,9 @@ func TestUnmarshal(t *testing.T) {
{testContentResource{key: "r1", content: `slogan = "Hugo Rocks!"`, mime: media.TOMLType}, nil, func(m map[string]interface{}) { {testContentResource{key: "r1", content: `slogan = "Hugo Rocks!"`, mime: media.TOMLType}, nil, func(m map[string]interface{}) {
assertSlogan(m) assertSlogan(m)
}}, }},
{testContentResource{key: "r1", content: `<root><slogan>Hugo Rocks!</slogan></root>"`, mime: media.XMLType}, nil, func(m map[string]interface{}) {
assertSlogan(m)
}},
{testContentResource{key: "r1", content: `1997,Ford,E350,"ac, abs, moon",3000.00 {testContentResource{key: "r1", content: `1997,Ford,E350,"ac, abs, moon",3000.00
1999,Chevy,"Venture ""Extended Edition""","",4900.00`, mime: media.CSVType}, nil, func(r [][]string) { 1999,Chevy,"Venture ""Extended Edition""","",4900.00`, mime: media.CSVType}, nil, func(r [][]string) {
c.Assert(len(r), qt.Equals, 2) c.Assert(len(r), qt.Equals, 2)