Commit graph

44 commits

Author SHA1 Message Date
Dietrich Epp
d5542ed286 deploy: Add stripIndexHtml target option
This new configuration parameter causes paths matching
"<dir>/index.html" to be stored as "<dir>/" remotely. This simplifies
the cloud configuration needed for some use cases, such as CloudFront
distributions with S3 bucket origins. Before this change, users must
configure their S3 buckets as public websites (which is incompatible
with certain authentication / authorization schemes), or users must add
a CloudFormation function to add index.html to the end of incoming
requests. After this change, users can simply use an ordinary CloudFront
distribution (no additional code) with an ordinary S3 bucket origin (and
not an S3 website).

This adds tests to ensure that functionality like matchers is unaffected
by this change. I have also tested that the functionality works as
expected when deploying to a real S3 / CloudFront website.

Closes #12607
2024-06-20 19:37:22 +02:00
Bjørn Erik Pedersen
0257eb50a4 Avoid impporting deploy from config when nodeploy tag is set
Test:

```
go list -tags nodeploy ./... | grep deploy
```

Fixes #12009
2024-02-07 20:38:13 +01:00
Bjørn Erik Pedersen
d8c2734178 deploy: Fix CloudFront invalidation with AWS SDK2 2024-01-31 22:12:48 +01:00
Franky W
a1c64989df Upgrade to deploy to use AWS SDK V2 2024-01-31 22:12:48 +01:00
Bjørn Erik Pedersen
a795acbcd8 all: Run gofumpt -l -w . 2024-01-28 23:14:09 +01:00
Bjørn Erik Pedersen
7285e74090
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaning
There are some breaking changes in this commit, see #11455.

Closes #11455
Closes #11549

This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build.

The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate.

A list of the notable new features:

* A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server.
* A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently.
* You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs.
We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections).
Memory Limit
* Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory.

New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively.

This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive):

Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get
Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers.
Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds.

Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role).

Fixes #10169
Fixes #10364
Fixes #10482
Fixes #10630
Fixes #10656
Fixes #10694
Fixes #10918
Fixes #11262
Fixes #11439
Fixes #11453
Fixes #11457
Fixes #11466
Fixes #11540
Fixes #11551
Fixes #11556
Fixes #11654
Fixes #11661
Fixes #11663
Fixes #11664
Fixes #11669
Fixes #11671
Fixes #11807
Fixes #11808
Fixes #11809
Fixes #11815
Fixes #11840
Fixes #11853
Fixes #11860
Fixes #11883
Fixes #11904
Fixes #7388
Fixes #7425
Fixes #7436
Fixes #7544
Fixes #7882
Fixes #7960
Fixes #8255
Fixes #8307
Fixes #8863
Fixes #8927
Fixes #9192
Fixes #9324
2024-01-27 16:28:14 +01:00
Matt Brown
c3f273b2d7 deploy: Create AWS session for CloudFront invalidation via Go CDK
This allows the AWS credentials to be picked up from the configured
target URL (like blob does) rather than the current behaviour of only
relying on the defaults.

Relying on the defaults here means having to specify credentials twice
(once in the URL for the blob, once in the environment for this code
path) when non-default AWS credentials are in used (e.g. via a profile).
2023-08-30 16:38:20 +03:00
Oleksandr Redko
d979831dbd deploy: Update InvalidateCloudFront to use Go CDK helper 2023-08-21 16:57:41 +02:00
Bjørn Erik Pedersen
5b4bfc2db7 Fix broken nodeploy setup
Fixes #11149
2023-06-23 12:40:43 +02:00
Bjørn Erik Pedersen
12dc9a6e4a deploy: Fix deploy defaults for non-zero flag values (e.g. maxDeletes, invalidateCDN)
This was broken in the config rewrite in Hugo 0.112.0.

The workaround is to be explicit about setting these flag values (even if just using the defaults), e.g.:

```
hugo deploy --invalidateCDN --maxDeletes 256
```

Fixes #11127
2023-06-18 19:38:34 +02:00
Bjørn Erik Pedersen
7c9fada778 Replace the old log setup, with structured logging etc.
Fixes #11124
2023-06-18 13:03:04 +02:00
Bjørn Erik Pedersen
241b21b0fd Create a struct with all of Hugo's config options
Primary motivation is documentation, but it will also hopefully simplify the code.

Also,

* Lower case the default output format names; this is in line with the custom ones (map keys) and how
it's treated all the places. This avoids doing `stringds.EqualFold` everywhere.

Closes #10896
Closes #10620
2023-05-16 18:01:29 +02:00
davidejones
873be9f90a ensure we default to 10 correctly 2023-03-07 15:38:26 +01:00
davidejones
bebb2b8d0a switch transfers to workers 2023-03-07 15:38:26 +01:00
davidejones
e6f029bdee customize parallel transfer count 2023-03-07 15:38:26 +01:00
Oleksandr Redko
d453c12742 Replace deprecated ioutil with io and os
https://pkg.go.dev/io/ioutil is deprecated since Go 1.16.
2023-03-01 16:28:43 +01:00
Anthony Fok
534e7155bb deps: Update to github.com/pelletier/go-toml/v2 v2.0.1 2022-06-06 09:36:43 +02:00
Bjørn Erik Pedersen
f2946da9e8 Improve error messages, esp. when the server is running
* Add file context to minifier errors when publishing
* Misc fixes (see issues)
* Allow custom server error template in layouts/server/error.html

To get to this, this commit also cleans up and simplifies the code surrounding errors and files. This also removes the usage of `github.com/pkg/errors`, mostly because of https://github.com/pkg/errors/issues/223 -- but also because most of this is now built-in to Go.

Fixes #9852
Fixes #9857
Fixes #9863
2022-05-06 19:43:22 +02:00
David Jones
d0657a436e
deploy: Set an MD5 attribute and use that if eTag not available
During Hugo deploy when a remote MD5 is invalid (e.g due to multipart eTag) Hugo reads the entire remote file and calculates the MD5 again which can be slow.

This commit updates the file upload so that it will also store an MD5 hash in the cloud provider's attributes. e.g in AWS this looks like x-amz-meta-md5chksum: 26fe392386a8123bf8956a16e08cb841.
2022-04-05 10:42:54 +02:00
Bjørn Erik Pedersen
b80853de90
all: gofmt -w -r 'interface{} -> any' .
Updates #9687
2022-03-17 22:03:27 +01:00
Joe Mooring
657d0272e3 Remove debug statement 2021-12-11 22:59:36 +01:00
Joe Mooring
f122771fb1 Check for empty deployment targets and matchers
Fixes #9220
2021-12-01 10:17:42 +01:00
Bjørn Erik Pedersen
d392893cd7
Misc config loading fixes
The main motivation behind this is simplicity and correctnes, but the new small config library is also faster:

```
BenchmarkDefaultConfigProvider/Viper-16         	  252418	      4546 ns/op	    2720 B/op	      30 allocs/op
BenchmarkDefaultConfigProvider/Custom-16        	  450756	      2651 ns/op	    1008 B/op	       6 allocs/op
```

Fixes #8633
Fixes #8618
Fixes #8630
Updates #8591
Closes #6680
Closes #5192
2021-06-14 17:00:32 +02:00
Robert van Gent
f523e9f0fd deploy: Don't treat a NotFound response for Delete as a fatal error. 2021-04-29 07:07:01 +02:00
Bjørn Erik Pedersen
ba1d0051b4 media: Make Type comparable
So we can use it and output.Format as map key etc.

This commit also fixes the media.Type implementation so it does not need to mutate itself to handle different suffixes for the same MIME type, e.g. jpg vs. jpeg.

This means that there are no Suffix or FullSuffix on media.Type anymore.

Fixes #8317
Fixes #8324
2021-03-14 15:21:54 +01:00
Bjørn Erik Pedersen
d90e37e0c6 all: Format code with gofumpt
See https://github.com/mvdan/gofumpt
2020-12-03 13:12:58 +01:00
José Moreira
56a343507c deploy: Do not call CDN service invalidation when executing a dry run deployment
Currently, if cache invalidation is enabled when running `hugo deploy --dryRun` with , the deployer does not take the dry run flag into consideration and triggers CloudFront/GoogleCloudCDN cache invalidation if any of those services are configured.

This change will instead print a feedback message if a dry run is in effect and quiet mode is not enabled.

Fixes #7884
2020-10-28 11:19:40 +01:00
Eric Hagman
f465c5c307
build: Allow optional "nodeploy" tag to exclude deploy command from bin
Fixes #7826
2020-10-23 09:03:41 +02:00
satotake
12f6a1cdc0
Respect mediatypes for deploy
Fixes #6861
2020-08-03 19:06:18 +02:00
John Weldon
558c09305e deploy: Do not suppress .well-known/ directory
Deployments ignore directories with a leading `.`, but should not ignore
certain well known 'hidden' directories like `.well-known/`

Fixes #6691
2020-05-14 09:56:53 +02:00
David Jones
51e178a6a2
deploy: Add include and exclude support for remote 2020-03-08 17:35:32 +01:00
Robert van Gent
05a74eaec0
deploy: Implement include/exclude filters for deploy
Fixes #6922
2020-02-27 07:26:05 +01:00
Garret Kelly
674e81ae87 deploy: Add ability to invalidate Google Cloud CDN 2019-10-03 16:53:45 +02:00
Fazal Majid
c0d7188ec8 deploy: Do not compile in Azure on Solaris
See #6324
2019-09-13 14:53:36 +02:00
Bjørn Erik Pedersen
9e57182705
tests: Convert from testify to quicktest 2019-08-12 13:26:32 +02:00
Christian Muehlhaeuser
e88d798990 Fixed tautological error conditions
Drop error & nil checks where the value can not have changed.
2019-08-10 01:04:43 +02:00
Robert van Gent
9df57154ee deploy: Default --target to the first deployment target 2019-06-08 00:41:59 +02:00
Robert van Gent
b0f536fb27 Drop dashes in http header matcher attributes; other changes from code review 2019-05-31 22:08:12 +02:00
Robert van Gent
5e83f42562 deploy: remove TODO comment about subfolders; handled by GoCDK blob URLs 2019-05-30 22:42:44 +02:00
Robert van Gent
5dc6d0df94 deploy: Add tests; fix Windows 2019-05-09 22:11:23 +02:00
Robert van Gent
527cf1ab03 deploy: Support configuration of upload order 2019-05-06 22:59:07 +02:00
Robert van Gent
f4956d9aae deploy: Support invalidating a CloudFront CDN cache 2019-05-06 21:09:33 +02:00
Robert van Gent
f330e869e4 deploy: compute MD5 by reading if List didn't provide one 2019-05-04 09:31:34 +02:00
Robert van Gent
c7165589b3 Add a "deploy" command 2019-05-03 17:58:40 +02:00