This also includes a refactor of the hugofs package and its usage.
The motivation for that is:
The Afero filesystems are brilliant. Hugo's way of adding a dozen of global variables for the different filesystems was a mistake. In readFile (and also in some other places in Hugo today) we need a way to restrict the access inside the working dir. We could use ioutil.ReadFile and implement the path checking, checking the base path and the dots ("..") etc. But it is obviously better to use an Afero BasePathFs combined witha ReadOnlyFs. We could create a use-once-filesystem and handle the initialization ourselves, but since this is also useful to others and the initialization depends on some other global state (which would mean to create a new file system on every invocation), we might as well do it properly and encapsulate the predefined set of filesystems. This change also leads the way, if needed, to encapsulate the file systems in a struct, making it possible to have several file system sets in action at once (parallel multilanguage site building? With Moore's law and all...)
Fixes#1551
This uses the Emoji map from https://github.com/kyokomi/emoji -- but with a custom replacement implementation.
The built-in are fine for most use cases, but in Hugo we do care about pure speed.
The benchmarks below are skewed in Hugo's direction as the source and result is a byte slice,
Kyokomi's implementation works best with strings.
Curious: The easy-to-use `strings.Replacer` is also plenty fast.
```
BenchmarkEmojiKyokomiFprint-4 20000 86038 ns/op 33960 B/op 117 allocs/op
BenchmarkEmojiKyokomiSprint-4 20000 83252 ns/op 38232 B/op 122 allocs/op
BenchmarkEmojiStringsReplacer-4 100000 21092 ns/op 17248 B/op 25 allocs/op
BenchmarkHugoEmoji-4 500000 5728 ns/op 624 B/op 13 allocs/op
```
Fixes#1891
Started to increase coverage in helpers package, now at 74.9% of statements.
In the process, also a few minor changes have been applied to content.go.
* Content.go has undergone a formatting refactor regarding comments
* Unused function TruncateWords has been removed
* RenderingContext's "mmark" has been changed to use MmarkRender
* Content_test.go added to cover content.go's functionality
especially when the given `--source` path is a relative directory.
Also, when `--source` is specified, make WorkingDir an absolute path
from the very beginning, to be consistent with the case when `--source`
is not given. Otherwise, the function name helpers.AbsPathify(), which
prepends WorkingDir to a relative path, does not really make sense.
Fixes#1721
To allow the end users to disable any form of smart dashes
(LaTeX-style or not) while keeping the rest of Blackfriday
SmartyPants features.
Depends on https://github.com/russross/blackfriday/pull/190
"Add HTML_SMARTYPANTS_DASHES for toggling smart dashes"
to be accepted by Blackfriday developers.
Inserts a code tag into Pygments output with the language-info that is present when using client-side highlighting (useful for CSS hooks)
```html
<code class="language-go" data-lang="go">
```
closes#1490
* add global `hasCJKLanguage` flag, if true, turn on auto-detecting CJKLanguage
* add `isCJKLanguage` frontmatter to force specify whether is CJKLanguage or not
* For .Summary: If isCJKLanguage is true, use the runes as basis for truncation, else keep as today.
* For WordCount: If isCJKLanguage is true, use the runes as basis for calculation, else keep as today.
* Unexport RuneCount
Fixes#1377
It currently handles --baseUrl to --baseURL, and --uglyUrls to --uglyURLs.
Special thanks to Eric Paris (@eparis) for writing the
"normalized name" support in Cobra, and for showing us
how it is used in Kubernetes.
See Issue #959
This simplifies the retrieval of the HTML (no more need to extract the
part within body) and also removes the unwanted "Last Updated" part in
the article.
This commit adds a new PygmentsCodeFences config option (default false), which if true will allow GitHub style backtick code fences around code, which will then be rendered by Pygments.
For example:
``` language
your code
```
can be used instead of {{< highlight language >}}your code {{< /highlight >}}.
Fixes#362
Previously if no language was specified, then illegal args would be passed to pygments, for example `pygments -l -fhtml`, which would result in pygments printing an error.
Reduce duplication (`x + FilePathSeparator + y` a few lines away from `filepath.Join(x, y)`) and add a `GetThemeDir()` function to get the current theme's directory.
Also add a comment complaining about the `GetThemesDirPath()` function, which doesn't seem to do what its name would suggest. This might be a candidate for deprecation?
So the taxonomy `Gérard Depardieu` gives paths on the form `gerard-depardieu`.
Unfortunately this introduces two imports from the `golang.org/`, but Unicode-normalization isn't something we'd want to write from scratch.
See https://blog.golang.org/normalization
See #1180
Section names are also used as the title of the list pages, but naming section folders as `Fish and Chips` and similar didn't work very well.
This commit fixes that.
This commit also changes the title casing of the section titles. Some may argue that this is a breaking change, but the old behaviour was also pretty broken,
even for languages that use title capitalizations, as it didn't follow any particular style guide, `fish and chips` became `Fish And Chips` etc.
Now it just turns the first letter into upper case, so `Fish and Chips` will be left as `Fish and Chips`.
People wanting the good old behaviour can use the `title` template func.
Fixes#1176
Setting `RelativeURLs` to `true` will make all relative URLs in the site *really* relative.
And will do so with speed.
So:
In `/post/myblogpost.html`:
`/mycss.css` becomes `../mycss.css`
The same in `/index.html` will become:
`./mycss.css` etc.
Note that absolute URLs will not be touched (either external resources, or URLs constructed with `BaseURL`).
The speediness is about the same as before:
```
benchmark old ns/op new ns/op delta
BenchmarkAbsURL 17462 18164 +4.02%
BenchmarkAbsURLSrcset 18842 19632 +4.19%
BenchmarkXMLAbsURLSrcset 18643 19313 +3.59%
BenchmarkXMLAbsURL 9283 9656 +4.02%
benchmark old allocs new allocs delta
BenchmarkAbsURL 24 28 +16.67%
BenchmarkAbsURLSrcset 29 32 +10.34%
BenchmarkXMLAbsURLSrcset 27 30 +11.11%
BenchmarkXMLAbsURL 12 14 +16.67%
benchmark old bytes new bytes delta
BenchmarkAbsURL 3154 3404 +7.93%
BenchmarkAbsURLSrcset 2376 2573 +8.29%
BenchmarkXMLAbsURLSrcset 2569 2763 +7.55%
BenchmarkXMLAbsURL 1888 1998 +5.83%
```
Fixes#1104Fixes#622Fixes#937Fixes#157
Many minor fixes to make test logs more consistent and correct a
mispelling.
Standardize on "[%i] got X but expected Y" for log messages. Using
a consistent layout makes it easier to read the test results. This
was mostly changing "Got" to "got". Swapped the order of values on
several calls to bring them in line with the convention.
A few log messages had a sequence number added to identify the
exact scenario that failed. Otherwise, there would be no way to
ascertain which failed When there are many scenarios.
Correct spelling of "expected."
Fixes#1028
Merged be2097e1ad
[close#1040]
Fixes part of issue 1030. Previously hugo only listed the content
directory as being watched. Now we list all files being watched
according to `commands.getDirList()`. We also introduce a RemoveSubpaths
function and test in the helpers module to reduce noise in the command
line output by not showing subdirectories of ones already being watched.
For example, instead of:
`Watching for changes in $HOME/blog/content`
We get:
`Watching for changes in
$HOME/blog/{data,content,layouts,static,themes/my-theme}`
`Paginate`now returns error when
1) `.Paginate` is called after `.Paginator`
2) `.Paginate` is repeatedly called with different arguments
This should help remove some confusion.
This commit also introduces DistinctErrorLogger, to prevent spamming the log for duplicate rendering errors from the pagers.
Fixes#993
As views looks like a regular template, but doesn't need a base template, we have to look inside it.
Altough really not needed by this commit, reading the full file content into memory just to do a substring search is a waste.
So this commit implements a `ReaderContains` func that in most cases should be much faster than doing an `ioutil.ReadAll` and `bytes.Contains`:
```
benchmark old ns/op new ns/op delta
BenchmarkReaderContains 78452 20260 -74.18%
benchmark old allocs new allocs delta
BenchmarkReaderContains 46 20 -56.52%
benchmark old bytes new bytes delta
BenchmarkReaderContains 46496 1258 -97.29%
```
Fixes#999
Put version handling into the helpers package so it can be used by many,
and split version and suffix to make it possible to calculate the next Hugo version.
Thanks to @bep's new, brilliant helpers.Deprecated() function,
the following functions or variables are transitioned to their
new names, preserving backward compatibility for v0.14
and warning the user of upcoming obsolescence in v0.15:
* .Url → .URL (for node, menu and paginator)
* .Site.BaseUrl → .Site.BaseURL
* .Site.Indexes → .Site.Taxonomies
* .Site.Recent → .Site.Pages
* getJson → getJSON
* getCsv → getCSV
* safeHtml → safeHTML
* safeCss → safeCSS
* safeUrl → safeURL
Also fix related initialisms in strings and comments.
Continued effort in fixing #959.
See #470
* Based on existing support for reStructuredText files
* Handles content files with extensions `.asciidoc` and `.ad`
* Pipes content through `asciidoctor --safe -`.
If `asciidoctor` is not installed, then `asciidoc --safe -`.
* To make sure `asciidoctor` or `asciidoc` is found, after adding
a piece of AsciiDoc content, run `hugo` with the `-v` flag
and look for this message:
INFO: 2015/01/23 Rendering with /usr/bin/asciidoctor ...
Caveats:
* The final "Last updated" timestamp is currently not stripped.
* When `hugo` is run with `-v`, you may see a lot of these messages
INFO: 2015/01/23 Rendering with /usr/bin/asciidoctor ...
if you have lots of `*.ad`, `*.adoc` or `*.asciidoc` files.
* Some versions of `asciidoc` may have trouble with its safe mode.
To test if you are affected, try this:
$ echo "Hello" | asciidoc --safe -
asciidoc: ERROR: unsafe: ifeval invalid
asciidoc: FAILED: ifeval invalid safe document
If so, I recommend that you install `asciidoctor` instead.
Feedback and patches welcome!
Ideally, we should be using https://github.com/VonC/asciidocgo,
@VonC's wonderful Go implementation of Asciidoctor. However,
there is still a bit of work needed for asciidocgo to expose
its API so that Hugo can actually use it.
Until then, hope this "experimental AsciiDoc support through external
helpers" can serve as a stopgap solution for our community. :-)
2015-01-30: Updated for the replaceShortcodeTokens() syntax change
2015-02-21: Add `.adoc` extension as suggested by @Fale
Conflicts:
helpers/content.go
First step to use initialisms that golint suggests,
for example:
Line 116: func GetHtmlRenderer should be GetHTMLRenderer
as see on http://goreportcard.com/report/spf13/hugo
Thanks to @bep for the idea!
Note that command-line flags (cobra and pflag)
as well as struct fields like .BaseUrl and .Url
that are used in Go HTML templates need more work
to maintain backward-compatibility, and thus
are NOT yet dealt with in this commit.
First step in fixing #959.
Make sure that the file separator is added to the temp dir in all cases.
This prevents cache temp files being written to the root temp folder.
Fixes#910
Temporary workaround for the bug fix and resulting
behavioral change in purell.NormalizeURLString():
a leading '/' was inadvertently to relative links,
but no longer, see #878.
I think the real solution is to allow Hugo to
make relative URL with relative path,
e.g. "../../post/hello-again/", as wished by users
in issues #157, #622, etc., without forcing
relative URLs to begin with '/'.
Once the fixes are in, let's remove this kludge
and restore SanitizeUrl() to the way it was.
Fixes#878
Allocates less memory:
benchmark old ns/op new ns/op delta
BenchmarkStripHTML 6572 6695 +1.87%
benchmark old allocs new allocs delta
BenchmarkStripHTML 5 4 -20.00%
benchmark old bytes new bytes delta
BenchmarkStripHTML 848 737 -13.09%
Compared to 0.12:
benchmark old ns/op new ns/op delta
BenchmarkStripHTML 10210 6695 -34.43%
benchmark old allocs new allocs delta
BenchmarkStripHTML 6 4 -33.33%
benchmark old bytes new bytes delta
BenchmarkStripHTML 1456 737 -49.38%
go test -test.run=NONE -bench=".*" -test.benchmem=true ./helpers
Old vs new impl (string.Replace vs string.Replacer):
benchmark old ns/op new ns/op delta
BenchmarkStripHTML 10210 6572 -35.63%
benchmark old allocs new allocs delta
BenchmarkStripHTML 6 5 -16.67%
benchmark old bytes new bytes delta
BenchmarkStripHTML 1456 848 -41.76%
The variable scope in the Go templates makes it hard, if possible at all, to write templates with counter variables or similar state.
This commit fixes that by adding a writable context to Node, backed by a map: Scratch.
This context has three methods, Get, Set and Add. The Add is tailored for counter variables, but can be used for any built-in numeric values or strings.
The paths were seen as changed but not static because of the backslashes in
ev.Name. Once the backslashes were added, I discovered that the JSON
sent to livereload was invalid and failed to work because it had backslashes.
Hence the code to replace the backslashes from the path to make them work
in JSON and for the URL.
With this fix, changes to a stylesheet are shown on the page, and if it's a
single file that changed, it's reflected in the browser without reloading the whole
page.
Two new configuration properties, `Paginate` (default `0`) and `PaginatePath` (default `page`) are added.
Setting `paginate` to a positive value will split the list pages for the home page, sections and taxonomies into chunks of size of the `paginate` property.
A `.Paginator` is provided to help building a pager menu.
There are two ways to configure a `.Paginator`:
1. The simplest way is just to call `.Paginator.Pages` from a template. It will contain the pages for "that page" (`.Data.Pages` will (like today) contain all the pages).
2. Select a sub-set of the pages with the available template functions and pass the slice to `.Paginate` : `{{ range (.Paginate (where .Data.Pages "Type" "post")).Pages }}`
**NOTE:** For a given Node, it's one of the options above. It's perfectly legitimate to iterate over the same pager more than once, but it's static and cannot change.
The `.Paginator` contains enough information to build a full-blown paginator interface.
The pages are built on the form (note: BLANK means no value, i.e. home page):
```
[SECTION/TAXONOMY/BLANK]/index.html
[SECTION/TAXONOMY/BLANK]/page/1/index.html => redirect to [SECTION/TAXONOMY/BLANK]/index.html
[SECTION/TAXONOMY/BLANK]/page/2/index.html
....
```
Fixes#96
Make Blackfriday's `HTML_SMARTYPANTS_FRACTIONS` option
user-configurable. Defaults to `true` as before. See
discussions at:
http://discuss.gohugo.io/t/any-way-to-disable-smart-fractions/328
Thanks to @bjornerik and @spf13 for laying the groundwork
making it easy to expose Blackfriday's underlying configurable
options.
canonifyUrls=true, RelPermalink and baseUrl with sub-path did not work.
This fixes that by adding a check for canonifyUrl=trues=true in RelPermalink().
So given
- baseUrl "http://somehost.com/sub/"
- the path "some-path/file.html"
For canonifyUrls=false RelPermalink() returns "/sub/some-path/file.html"
For canonifyUrls=true RelPermalink() returns "/some-path/file.html"
In the last case, the Url will be made absolute and clickable in a later step.
This commit also makes the menu urls defined in site config releative. To make them work with canonifying of urls, the context root is prepended if canonifying is turned off.
Fixes#519Fixes#711
(Experimental) reStructuredText support was working in v0.12,
but was no longer handled after some refactoring in v0.13-DEV.
That experimental support is now restored.
Furthermore, check for both rst2html and rst2html.py in the PATH,
and execute whichever is found.
See #472 for more information.
The flag `HTML_SMARTYPANTS_ANGLED_QUOTES` was added to Blackfriday on Black Friday. This configures rendering of double quotes as angled left and right quotes («
»).
Typical use cases would be either or, or combined, but never in the same
document. As an example would be a person from Norway; he has a blog in both
English and Norwegian (his native tongue); he would then configure Blackfriday
to use angled quotes for the Norwegian section, but keep them as reqular
double quotes for the English.
This commit adds configuration support for this new flag, configuration that can be set in the site configuration, but overridden in page front matter.
Fixes#605
File handling was broken on Windows. This commit contains a revision of the path handling with separation of file paths and urls where needed.
There may be remaining issues and there may be better ways to do this, but it is easier to start that refactoring job with a set of passing tests.
Fixes#687Fixes#660
Enable blackfriday.EXTENSION_AUTO_HEADER_IDS to generate the name of the
header ID from the text in the header. Works for prefix and underline
headers.
- TOC extraction had to be modified to look for `<li><a href="#`>
instead of `#toc_` because of this change.
- Fixed a number of tests that depended on the presence of `toc_` with
as an `id` or as a `href` value.
- Renames the earlier parameter `footnoteref` to `documentId` as it more
accurately represents the nature of the parameter. The `documentId` is
appended to all generated headers through the new HTML renderer
parameter `HeaderIDSuffix`.
This commit contains a restructuring and partial rewrite of the shortcode handling.
Prior to this commit rendering of the page content was mingled with handling of the shortcodes. This led to several oddities.
The new flow is:
1. Shortcodes are extracted from page and replaced with placeholders.
2. Shortcodes are processed and rendered
3. Page is processed
4. The placeholders are replaced with the rendered shortcodes
The handling of summaries is also made simpler by this.
This commit also introduces some other chenges:
1. distinction between shortcodes that need further processing and those who do not:
* `{{< >}}`: Typically raw HTML. Will not be processed.
* `{{% %}}`: Will be processed by the page's markup engine (Markdown or (infuture) Asciidoctor)
The above also involves a new shortcode-parser, with lexical scanning inspired by Rob Pike's talk called "Lexical Scanning in Go",
which should be easier to understand, give better error messages and perform better.
2. If you want to exclude a shortcode from being processed (for documentation etc.), the inner part of the shorcode must be commented out, i.e. `{{%/* movie 47238zzb */%}}`. See the updated shortcode section in the documentation for further examples.
The new parser supports nested shortcodes. This isn't new, but has two related design choices worth mentioning:
* The shortcodes will be rendered individually, so If both `{{< >}}` and `{{% %}}` are used in the nested hierarchy, one will be passed through the page's markdown processor, the other not.
* To avoid potential costly overhead of always looking far ahead for a possible closing tag, this implementation looks at the template itself, and is branded as a container with inner content if it contains a reference to `.Inner`
Fixes#565Fixes#480Fixes#461
And probably some others.
FileAndExt has now be completely rewritten and now works as may
reasonably be expected. The test cases for:
ReplaceExtension
Filename
FileAndExt
now all pass.
The problem was the way path.Base was being used.
Firstly Base returns "." if the directory is empty, but it can also
return "." for the current directory, or ".." for the parent directory,
if these are the last elements in the path. Simply detecting the
presence of a "." in the returned string and truncating before the "."
does not therefore always result in a valid filename.
Secondly, Base strips any trailing slashes making is more difficult to
detect when the path does not end in a filename but in a directory name
i.e. a no filename case. Not detecting this incorrectly results in the
last directory name being returned as the filename.
The code has been updated to take account of of both situations so that:
1) An empty string for both the filename and extension is returned if the
path does not contain a filename (plus an optional extension). This
includes both the empty path case, and the ends in a directory case.
2) If the path represents a filename with an extension then both the
filename and the extension (minus any dots) are returned as non empty
strings.
3) If the path represents a filename without an extension then filename
will be returned as a non empty string (minus any dot) and the extension
will be returned as an empty string.
Updated the test cases in TestReplaceExtension to reflect the intent
of ReplaceExtension which is to accept a path and return only the file
name with a new extension. It's intentionally designed to strip out the
path and only provide the filename
Additional test cases have been added to both TestFilename and
TestFileAndExt to clarify behaviour with relative paths.
Oops, my bad.
Despite the function intending to test SafeWriteToDisk it was actually
calling WriteToDisk. WriteToDisk does not return the file exists error
that SafeWriteToDisk does, which the test checks for.
Added the new path modules test file. This replaces the old
helpers_test.go file.
The currently failing tests are:
TestReplaceExtension
TestFilename
TestFileAndExt
TestGuessSection
TestFindCWD
TestWriteToDisk
In addition the TestSafeWriteToDisk test case is currently disabled.
It will panic if enabled.
In addition there are some minor changes to path.go. They are:
Refactored MakePathToLower to simplify it.
Commented out, pending removal, Sanitize as it appears to be unused.
Fixed the resource leak in UnicodeSanitize
Conflicts:
helpers/path.go
Added the new url module test file. This replaces the original
helpers_test.go file.
The TestUrlPrep test currently fails.
The only minor change to url.go is to add some trace printf's to
UrlPrep.
Added the new general module's test file, general_test.go. This replaces the
helpers_test.go file.
There is also a minor defect fix in general.go's StripHTML function.
The correct xhtml tag for a break is <br /> not </br>. I've also removed
the unnecessary spaces before the replacement "\n".
The new test module also reflects this change.
Conflicts:
helpers/general.go
Prior to this commit a dummy JavaScript filename was sent to LiveReload when changing a static file (CSS, image etc.), forcing a full browser reload of the page.
This commit fixes this by sending the relative file path of the changed static resource, enabling partial live reloading for CSS- and image-changes. If more than one static file happens to end up in the same changeevent-batch, it will fall back to do a full refresh. To enable this logic, the change events with names ending with ".goutputstream*" is now filtered out as temporary.
Changes in dynamic content behaves like before.
Issue #490
Mainly this was a change to helpers.MakePermalink, but to get the local server to run correctly,
we needed to redirect the path of the request from /foo to /. In addition, I added tests for the
server's code for fixing up the base url with different config file & CLI options.
