Commit graph

860 commits

Author SHA1 Message Date
Bjørn Erik Pedersen 4e14cf7607
Fail with error when double-rendering text in markdownify/RenderString
This commit prevents the most commons case of infinite recursion in link render hooks when the `linkify` option is enabled (see below). This is always a user error, but getting a `stack overflow` (the current stack limit in Go is 1 GB on 64-bit, 250 MB on 32-bit) error isn't very helpful. This fix will not prevent all such errors, though, but we may do better once #9570 is in place.

So, these will fail:

```
<a href="{{ .Destination | safeURL }}" >{{ .Text | markdownify }}</a>
<a href="{{ .Destination | safeURL }}" >{{ .Text | .Page.RenderString }}</a>
```

`.Text` is already rendered to `HTML`. The above needs to be rewritten to:

```
<a href="{{ .Destination | safeURL }}" >{{ .Text | safeHTML }}</a>
<a href="{{ .Destination | safeURL }}" >{{ .Text | safeHTML }}</a>
```

Fixes #8959
2022-03-10 08:19:03 +01:00
Joe Mooring f98e570b17 Add lang attribute to internal alias template
Closes #9586
2022-03-09 19:06:59 +01:00
Bjørn Erik Pedersen 4576c82ed4 Cache reflect.MethodByName
The isolated benchmark for the function is obviously much faster:

```bash
name                old time/op    new time/op    delta
GetMethodByName-10    1.21µs ± 7%    0.23µs ± 5%   -81.42%  (p=0.029 n=4+4)

name                old alloc/op   new alloc/op   delta
GetMethodByName-10      680B ± 0%        0B       -100.00%  (p=0.029 n=4+4)

name                old allocs/op  new allocs/op  delta
GetMethodByName-10      20.0 ± 0%       0.0       -100.00%  (p=0.029 n=4+4)
```

But more pleasing is the overall performance looking at the site benchmarks:

```bash
name                                      old time/op    new time/op    delta
SiteNew/Regular_Bundle_with_image-10        6.25ms ± 2%    6.10ms ± 2%     ~     (p=0.057 n=4+4)
SiteNew/Regular_Bundle_with_JSON_file-10    6.30ms ± 2%    5.66ms ±11%     ~     (p=0.057 n=4+4)
SiteNew/Regular_Tags_and_categories-10      22.2ms ± 2%    17.4ms ± 1%  -21.88%  (p=0.029 n=4+4)
SiteNew/Regular_Canonify_URLs-10             108ms ± 0%     107ms ± 0%   -1.20%  (p=0.029 n=4+4)
SiteNew/Regular_Deep_content_tree-10        36.1ms ± 1%    33.8ms ± 1%   -6.44%  (p=0.029 n=4+4)
SiteNew/Regular_TOML_front_matter-10        24.9ms ± 1%    22.6ms ± 1%   -9.30%  (p=0.029 n=4+4)
SiteNew/Regular_Many_HTML_templates-10      17.9ms ± 1%    16.7ms ± 1%   -6.43%  (p=0.029 n=4+4)
SiteNew/Regular_Page_collections-10         23.3ms ± 1%    22.0ms ± 0%   -5.58%  (p=0.029 n=4+4)
SiteNew/Regular_List_terms-10               8.00ms ± 1%    7.63ms ± 0%   -4.62%  (p=0.029 n=4+4)

name                                      old alloc/op   new alloc/op   delta
SiteNew/Regular_Bundle_with_image-10        2.10MB ± 0%    2.07MB ± 0%   -1.46%  (p=0.029 n=4+4)
SiteNew/Regular_Bundle_with_JSON_file-10    1.88MB ± 0%    1.85MB ± 0%   -1.76%  (p=0.029 n=4+4)
SiteNew/Regular_Tags_and_categories-10      13.5MB ± 0%    11.6MB ± 0%  -13.99%  (p=0.029 n=4+4)
SiteNew/Regular_Canonify_URLs-10            96.1MB ± 0%    95.8MB ± 0%   -0.40%  (p=0.029 n=4+4)
SiteNew/Regular_Deep_content_tree-10        28.4MB ± 0%    27.3MB ± 0%   -3.83%  (p=0.029 n=4+4)
SiteNew/Regular_TOML_front_matter-10        16.9MB ± 0%    15.1MB ± 0%  -10.58%  (p=0.029 n=4+4)
SiteNew/Regular_Many_HTML_templates-10      8.98MB ± 0%    8.44MB ± 0%   -6.04%  (p=0.029 n=4+4)
SiteNew/Regular_Page_collections-10         17.1MB ± 0%    16.5MB ± 0%   -3.91%  (p=0.029 n=4+4)
SiteNew/Regular_List_terms-10               3.92MB ± 0%    3.72MB ± 0%   -5.03%  (p=0.029 n=4+4)

name                                      old allocs/op  new allocs/op  delta
SiteNew/Regular_Bundle_with_image-10         25.8k ± 0%     24.9k ± 0%   -3.49%  (p=0.029 n=4+4)
SiteNew/Regular_Bundle_with_JSON_file-10     25.8k ± 0%     24.9k ± 0%   -3.49%  (p=0.029 n=4+4)
SiteNew/Regular_Tags_and_categories-10        288k ± 0%      233k ± 0%  -18.90%  (p=0.029 n=4+4)
SiteNew/Regular_Canonify_URLs-10              375k ± 0%      364k ± 0%   -2.80%  (p=0.029 n=4+4)
SiteNew/Regular_Deep_content_tree-10          314k ± 0%      283k ± 0%   -9.77%  (p=0.029 n=4+4)
SiteNew/Regular_TOML_front_matter-10          302k ± 0%      252k ± 0%  -16.55%  (p=0.029 n=4+4)
SiteNew/Regular_Many_HTML_templates-10        133k ± 0%      117k ± 0%  -11.81%  (p=0.029 n=4+4)
SiteNew/Regular_Page_collections-10           202k ± 0%      183k ± 0%   -9.55%  (p=0.029 n=4+4)
SiteNew/Regular_List_terms-10                55.6k ± 0%     49.8k ± 0%  -10.40%  (p=0.029 n=4+4)
```

Thanks to @quasilyte for the suggestion.

Fixes 9386
2022-03-08 19:36:55 +01:00
Bjørn Erik Pedersen 673cde1eb1 tpl/os: Revert readDir in theme behaviour
Fixes #9599
2022-03-04 08:43:47 +01:00
Bjørn Erik Pedersen 0327da050f tpl/transform: Fix it when template.HTML is passes as option to Hightlight
Fixes #9591
2022-03-02 12:30:35 +01:00
Bjørn Erik Pedersen 9b8b6d34e2 tpl/partials: Fix partialCached deadlock regression
This is a rollback of  0927cf739f

We cannot do that change until we either completes #9570 or possibly also use the new TryLock in GO 1.18.

Fixes #9588
Opens #4086
2022-03-02 11:16:21 +01:00
Bjørn Erik Pedersen 376704d382 tpl/collections: Fix apply when function have Context as first arg
As introduced in `partial` and `partialCached` in Hugo 0.93.0.

Fixes #9585
2022-03-01 13:18:02 +01:00
Bjørn Erik Pedersen 12d00d288c
docs: Regenerate docs helper 2022-02-28 08:59:57 +01:00
Bjørn Erik Pedersen fd0c1a5e9b tpl/diagrams: Rename the SVG accessor to Wrapped
Not perfect, but it gets it in line with the other .Inner/.Wrapped combos.
2022-02-27 19:51:40 +01:00
Bjørn Erik Pedersen 39261b689e tpl/transform: Add CanHighlight
Closes #9573
2022-02-27 19:51:40 +01:00
Bjørn Erik Pedersen 5f67151550 tpl/diagrams: Rename Body to Inner
This relates to #9577 – there's no concept a body element in SVG, so we might as well use an exsisting "Hugo term".
2022-02-27 17:59:36 +01:00
Bjørn Erik Pedersen f7109771a0 CodeblockContext method renames
Fixes #9577
2022-02-27 17:59:36 +01:00
Bjørn Erik Pedersen 10928a4f78 Remove the trailing new line in .Code
Fixes #9572
2022-02-26 21:54:36 +01:00
Bjørn Erik Pedersen 1a257bb2ba
Move the Goat template to the correct place
Updates #7765
2022-02-25 00:09:07 +01:00
Bjørn Erik Pedersen 08fdca9d93 Add Markdown diagrams and render hooks for code blocks
You can now create custom hook templates for code blocks, either one for all (`render-codeblock.html`) or for a given code language (e.g. `render-codeblock-go.html`).

We also used this new hook to add support for diagrams in Hugo:

* Goat (Go ASCII Tool) is built-in and enabled by default; just create a fenced code block with the language `goat` and start draw your Ascii diagrams.
* Another popular alternative for diagrams in Markdown, Mermaid (supported by GitHub), can also be implemented with a simple template. See the Hugo documentation for more information.

Updates #7765
Closes #9538
Fixes #9553
Fixes #8520
Fixes #6702
Fixes #9558
2022-02-24 18:59:50 +01:00
Akimo 6ff39fd908 Change disqus_config to window.disqus_config 2022-02-23 21:19:30 +01:00
Joe Mooring bddcfd911c deps: Update github.com/gohugoio/localescompressed v0.14.0 => v0.15.0
Fixes #9446
2022-02-19 11:34:00 +01:00
Bjørn Erik Pedersen 929808190f tpl/partials: Fix recently introduced deadlock in partials cache
The change in lock logic for `partialCached` in  0927cf739f was naive as it didn't consider cached partials calling other cached partials.

This changeset may look on the large side for this particular issue, but it pulls in part of a working branch, introducing `context.Context` in the template execution.

Note that the context is only partially implemented in this PR, but the upcoming use cases will, as one example, include having access to the top "dot" (e.g. `Page`) all the way down into partials and shortcodes etc.

The earlier benchmarks rerun against master:

```bash
name              old time/op    new time/op    delta
IncludeCached-10    13.6ms ± 2%    13.8ms ± 1%    ~     (p=0.343 n=4+4)

name              old alloc/op   new alloc/op   delta
IncludeCached-10    5.30MB ± 0%    5.35MB ± 0%  +0.96%  (p=0.029 n=4+4)

name              old allocs/op  new allocs/op  delta
IncludeCached-10     74.7k ± 0%     75.3k ± 0%  +0.77%  (p=0.029 n=4+4)
```

Fixes #9519
2022-02-17 18:47:36 +01:00
Bjørn Erik Pedersen 667f3a4ba8
tpl/partials: Add some more partials to BenchmarkIncludeCached 2022-02-17 16:45:59 +01:00
Bjørn Erik Pedersen 0927cf739f tpl/partials: Make sure a cached partial is invoked only once
This commit revises the locking strategy for `partialCached`. We have added a benchmark that may be a little artificial, but it should at least show that we're not losing any performance over this:

```bash
name              old time/op    new time/op    delta
IncludeCached-10    12.2ms ± 2%    11.3ms ± 1%   -7.36%  (p=0.029 n=4+4)

name              old alloc/op   new alloc/op   delta
IncludeCached-10    7.17MB ± 0%    5.09MB ± 0%  -29.00%  (p=0.029 n=4+4)

name              old allocs/op  new allocs/op  delta
IncludeCached-10      128k ± 1%       70k ± 0%  -45.42%  (p=0.029 n=4+4)
```

This commit also revises the template metrics hints logic a little, and add a test for it, which output is currently this:

```bash
 cumulative       average       maximum      cache  percent  cached  total
       duration      duration      duration  potential   cached   count  count  template
     ----------      --------      --------  ---------  -------  ------  -----  --------
      163.334µs     163.334µs     163.334µs          0        0       0      1  index.html
       23.749µs       5.937µs      19.916µs         25       50       2      4  partials/dynamic1.html
        9.625µs       4.812µs        6.75µs        100       50       1      2  partials/static1.html
        7.625µs       7.625µs       7.625µs        100        0       0      1  partials/static2.html
```

Some notes:

* The duration now includes the cached invocations (which should be very short)
* A cached template gets executed once before it gets cached, so the "percent cached" will never be 100.

Fixes #4086
Fixes #9506
2022-02-17 11:53:24 +01:00
Cameron Moore d16228334d metrics: Add cached count tracking
Track cached partial executions and display more useful info when
showing the hints output. Also group cache hints and counters together
to the right of the timing measurements.

Sample output:

     cumulative       average       maximum      cache  percent  cached  total
       duration      duration      duration  potential   cached   count  count  template
     ----------      --------      --------  ---------  -------  ------  -----  --------
       2.6973ms     674.325µs       857.3µs          0        0       0      4  _internal/_default/rss.xml
       2.6295ms     657.375µs       861.3µs          0        0       0      4  _default/single.html
       1.2563ms     314.075µs       381.5µs          0        0       0      4  news/single.html
        799.2µs       399.6µs       573.3µs          0        0       0      2  _internal/_default/sitemap.xml
        667.3µs       66.73µs       238.6µs         84        0       0     10  partials/header.html
        656.7µs      328.35µs       459.4µs          0        0       0      2  _default/list.html
        479.7µs       47.97µs         158µs         92        0       0     10  partials/head.html
        323.3µs       323.3µs       323.3µs          0        0       0      1  _internal/alias.html
        200.7µs       200.7µs       200.7µs          0        0       0      1  _internal/_default/sitemapindex.xml
         36.1µs       6.016µs        32.6µs        100       33       2      6  partials/footer.html
         27.4µs        13.7µs        26.7µs          0        0       0      2  index.html

Fixes #8375
2022-02-16 10:05:17 +01:00
Bjørn Erik Pedersen ea6bcd6944
tpl: Remove TODO comment 2022-02-15 23:39:54 +01:00
Bjørn Erik Pedersen f2e7b49acf Add --printUnusedTemplates
Fixes #9502
2022-02-15 20:01:57 +01:00
Bjørn Erik Pedersen b0a9cf0a75 tpl: Use go:embed to load internal templates
Fixes #8297
2022-02-14 09:39:46 +01:00
Bjørn Erik Pedersen da4866c2be Simplify some integration tests 2022-02-10 20:43:19 +01:00
Bjørn Erik Pedersen 6a238a7276 tpl/templates: Fix templates.Exist issue with base templates
Fixes #9477
2022-02-09 22:33:07 +01:00
Bjørn Erik Pedersen 39f69ca7f0 openapi3: Port integration test into correct package 2022-02-09 15:41:32 +01:00
Joe Mooring 9262719092 Validate comparison operator argument count
Fixes #9462
2022-02-05 17:41:43 +01:00
Bjørn Erik Pedersen 2655739940 tpl/collections: Fix apply with namespaced template funcs
We changed the signature to `func(...interface{}) (interface{}, error)` some time ago, but sadly we had no test for this for `apply`. Now we do.

Fixes #9393
2022-01-17 09:50:48 +01:00
Bjørn Erik Pedersen b84745d497
tpl/resources: Add empty method mapping for GetRemote
To get auto generated documentation for it.
2021-12-17 10:30:10 +01:00
Bjørn Erik Pedersen 44954497bc
Always use content to resolve content type in resources.GetRemote
This is a security hardening measure; don't trust the URL extension or any `Content-Type`/`Content-Disposition` header on its own, always look at the file content using Go's `http.DetectContentType`.

This commit also adds ttf and otf media type definitions to Hugo.

Fixes #9302
Fixes #9301
2021-12-17 09:50:28 +01:00
Bjørn Erik Pedersen 22ef5da20d
Add resources.GetRemote
In Hugo 0.89 we added remote support to `resources.Get`.

In hindsight that was not a great idea, as a poll from many Hugo users showed. See Issue #9285 for more details.

After this commit `resources.Get` only supports local resource lookups. If you want to support both, you need to use a construct similar to:

Also improve some option case handling.

```
{{ resource := "" }}
{{ if (urls.Parse $url).IsAbs }}
{{ $resource = resources.GetRemote $url }}
{{ else }}
{{ $resource = resources.Get $url }}
{{ end }}
```

Fixes #9285
Fixes #9296
2021-12-17 09:33:51 +01:00
Paul Gottschling 5758c370ea
Allow for return partials with falsy arguments (#9298)
Partials with returns values are parsed, then inserted into a
partial return wrapper via wrapInPartialReturnWrapper in order
to assign the return value via *contextWrapper.Set. The
predefined wrapper template for partials inserts a partial's nodes
into a "with" template action in order to set dot to a
*contextWrapper within the partial. However, because "with" is
skipped if its argument is falsy, partials with falsy arguments
were not being evaluated.

This replaces the "with" action in the partial wrapper with a
"range" action that isn't skipped if .Arg is falsy.

Fixes #7528
2021-12-17 08:35:21 +01:00
Bjørn Erik Pedersen f4389e48ce
Add some basic security policies with sensible defaults
This ommmit contains some security hardening measures for the Hugo build runtime.

There are some rarely used features in Hugo that would be good to have disabled by default. One example would be the "external helpers".

For `asciidoctor` and some others we use Go's `os/exec` package to start a new process.

These are a predefined set of binary names, all loaded from `PATH` and with a predefined set of arguments. Still, if you don't use `asciidoctor` in your project, you might as well have it turned off.

You can configure your own in the new `security` configuration section, but the defaults are configured to create a minimal amount of site breakage. And if that do happen, you will get clear instructions in the loa about what to do.

The default configuration is listed below. Note that almost all of these options are regular expression _whitelists_ (a string or a slice); the value `none` will block all.

```toml
[security]
  enableInlineShortcodes = false
  [security.exec]
    allow = ['^dart-sass-embedded$', '^go$', '^npx$', '^postcss$']
    osEnv = ['(?i)^(PATH|PATHEXT|APPDATA|TMP|TEMP|TERM)$']

  [security.funcs]
    getenv = ['^HUGO_']

  [security.http]
    methods = ['(?i)GET|POST']
    urls = ['.*']
```
2021-12-16 09:40:22 +01:00
Bjørn Erik Pedersen 3bc6830411 Remove the retries on error in remote resources.Get
Fixes #9271
See  #9259
2021-12-10 11:10:41 +01:00
Bjørn Erik Pedersen e4d6ec94b5 Allow user to handle/ignore errors in resources.Get
In Hugo 0.90.0 we introduced remote support in `resources.Get`.

But with remote resources comes with a higher chance of failing a build (network issues, remote server down etc.).

Before this commit we always failed the build on any unexpected error.

This commit allows the user to check for any error (and potentially fall back to a default local resource):

```htmlbars
{{ $result := resources.Get "https://gohugo.io/img/hugo-logo.png" }}
{{ with $result }}
        {{ if .Err }}
        {{/* log the error, insert a default image etc. *}}
        {{ else }}
        <img src="{{ .RelPermalink }}" width="{{ .Width }}" height="{{ .Height }}" alt="">
        {{ end }}
{{ end }}
```

Note that the default behaviour is still to fail the build, but we will delay that error until you start using the `Resource`.

Fixes #9529
2021-12-10 11:10:41 +01:00
Joe Mooring 5538507e90 tpl/transform: Optional options for highlight func
Closes #9249
Fixes gohugoio/hugoDocs#63
2021-12-07 11:26:56 +01:00
Paul van Brouwershaven 0eaaa8fee3
Implement XML data support
Example:

```
{{ with resources.Get "https://example.com/rss.xml" | transform.Unmarshal }}
    {{ range .channel.item }}
        <strong>{{ .title | plainify | htmlUnescape }}</strong><br />
        <p>{{ .description | plainify | htmlUnescape }}</p>
        {{ $link := .link | plainify | htmlUnescape }}
        <a href="{{ $link }}">{{ $link }}</a><br />
        <hr>
    {{ end }}
{{ end }}
```

Closes #4470
2021-12-02 17:30:36 +01:00
Bjørn Erik Pedersen 93572e5318 resources: Add timeout to the HTTP request in Get
Workaround for https://github.com/golang/go/issues/49366
2021-12-02 16:11:14 +01:00
Joe Mooring 133e4bfbee Remove empty href element from pagination template
Closes #9149
2021-12-02 09:14:30 +01:00
Paul van Brouwershaven 8aa7257f65
Add remote support to resources.Get
Closes #5255
Supports #9044
2021-11-30 11:49:51 +01:00
Joe Mooring 75a823a36a Add deprecation warning to google_news template
Closes #9172
2021-11-21 11:25:17 +01:00
Joe Mooring 04a3b45db4 Fix description of lang.FormatNumberCustom
It currently refers to itself as a simple alternative, when it should
refer to lang.FormatNumber.
2021-11-01 18:54:43 +01:00
Joe Mooring 0cc39af682 Update Twitter shortcode oEmbed endpoint
The existing endpoint will be retired and removed on November 23, 2021.
References:

- https://twittercommunity.com/t/consolidating-the-oembed-functionality/154690
- https://developer.twitter.com/en/docs/twitter-for-websites/oembed-api#Embedded

This is a backward compatible change.

The existing endpoint requires a single parameter: the id of the tweet.

The new endpoint requires two parameters: the id of the tweet, and the
user with whom it is associated. For the moment, if you supply the wrong
user, the request will be redirected (with a small delay) to the correct
user/id pair. This behavior is undocumented, but we will take advantage
of it as Hugo site authors transition to the new syntax.

{{< tweet 1453110110599868418 >}} --> works, throws warning, deprecate at some point

{{< tweet user="SanDiegoZoo" id="1453110110599868418" >}} --> new syntax

Fixes #8130
2021-11-01 15:51:00 +01:00
Bjørn Erik Pedersen e82cbd746f
tpl/time: Use configured location when date passed to Format is string
Updates #9084
2021-10-30 16:09:38 +02:00
Brad e55466ce70
tpl/path: Add path.Clean
Fixes #8885
2021-10-05 16:15:10 +02:00
Paul Gottschling a864ffe9ac Clarify "precision" in currency format functions
The documentation of the FormatAccounting and FormatCurrency
functions could be clearer in terms of how the precision param
works. This commit makes it more explicit that adding a precision
of < 2 will not format the return values to include fewer decimals.

Resolves #8858
2021-09-22 20:00:30 +02:00
Bjørn Erik Pedersen abd969a670
Revert "tpl/time: Handle nil values in time.AsTime"
This reverts commit 3e11072892.
2021-08-10 13:21:49 +02:00
Bjørn Erik Pedersen 3e11072892
tpl/time: Handle nil values in time.AsTime
Closes #8865
2021-08-09 18:44:35 +02:00
Bjørn Erik Pedersen 9bba9a3a98
parser: Indent TOML tables
Fixes #8850
2021-08-04 11:39:16 +02:00