Commit graph

76 commits

Author SHA1 Message Date
Bjørn Erik Pedersen 6636cf1bea
Resolve error handling/parser related TODOs
See #5324
2018-10-23 19:41:22 +02:00
Bjørn Erik Pedersen f669ef6bec
herrors: Improve handling of JSON errors
`*json.UnmarshalTypeError` and `*json.SyntaxError` has a byte `Offset`, so use that.

This commit also reworks/simplifies the errror line matching logic. This also makes the file reading unbuffered, but that should be fine in this error case.

See #5324
2018-10-23 14:35:43 +02:00
Bjørn Erik Pedersen d1661b823a
hugolib: Continue the file context/line number errors work
See #5324
2018-10-22 20:46:14 +02:00
Bjørn Erik Pedersen 7930d2132a
hugolib: Remove the now superflous Source struct
See #5324
2018-10-22 20:46:14 +02:00
Bjørn Erik Pedersen eb038cfa0a
Convert the rest to new page parser code paths
And remove some now unused code.

See #5324
2018-10-22 20:46:14 +02:00
Bjørn Erik Pedersen 129c27ee6e
parser/metadecoders: Consolidate the metadata decoders
See #5324
2018-10-22 20:46:13 +02:00
Bjørn Erik Pedersen 44da60d869
hugolib: Redo the summary delimiter logic
Now that we have a proper page parse tree, this can be greatly simplified.

See #5324
2018-10-22 20:46:13 +02:00
Bjørn Erik Pedersen 1e3e34002d
hugolib: Integrate new page parser
See #5324
2018-10-22 20:46:13 +02:00
Bjørn Erik Pedersen 1b7ecfc2e1
hugolib: Use []byte in shortcode parsing
See #5324
2018-10-22 19:57:44 +02:00
Bjørn Erik Pedersen 27f5a906a2
parser/pageparser: Use []byte in page lexer
See #5324
2018-10-22 19:57:44 +02:00
Bjørn Erik Pedersen 2fdc4a24d5
parser/pageparser: Add front matter etc. support
See #5324
2018-10-22 19:57:43 +02:00
Bjørn Erik Pedersen f6863e1ef7
parser/pageparser: File renames and splitting
See #5324
2018-10-22 19:57:43 +02:00
Bjørn Erik Pedersen d6c16afde0
Move the shortcode parser to the new pageparser package
See #5324
2018-10-22 19:57:43 +02:00
Bjørn Erik Pedersen dea71670c0
Add Hugo Piper with SCSS support and much more
Before this commit, you would have to use page bundles to do image processing etc. in Hugo.

This commit adds

* A new `/assets` top-level project or theme dir (configurable via `assetDir`)
* A new template func, `resources.Get` which can be used to "get a resource" that can be further processed.

This means that you can now do this in your templates (or shortcodes):

```bash
{{ $sunset := (resources.Get "images/sunset.jpg").Fill "300x200" }}
```

This also adds a new `extended` build tag that enables powerful SCSS/SASS support with source maps. To compile this from source, you will also need a C compiler installed:

```
HUGO_BUILD_TAGS=extended mage install
```

Note that you can use output of the SCSS processing later in a non-SCSSS-enabled Hugo.

The `SCSS` processor is a _Resource transformation step_ and it can be chained with the many others in a pipeline:

```bash
{{ $css := resources.Get "styles.scss" | resources.ToCSS | resources.PostCSS | resources.Minify | resources.Fingerprint }}
<link rel="stylesheet" href="{{ $styles.RelPermalink }}" integrity="{{ $styles.Data.Digest }}" media="screen">
```

The transformation funcs above have aliases, so it can be shortened to:

```bash
{{ $css := resources.Get "styles.scss" | toCSS | postCSS | minify | fingerprint }}
<link rel="stylesheet" href="{{ $styles.RelPermalink }}" integrity="{{ $styles.Data.Digest }}" media="screen">
```

A quick tip would be to avoid the fingerprinting part, and possibly also the not-superfast `postCSS` when you're doing development, as it allows Hugo to be smarter about the rebuilding.

Documentation will follow, but have a look at the demo repo in https://github.com/bep/hugo-sass-test

New functions to create `Resource` objects:

* `resources.Get` (see above)
* `resources.FromString`: Create a Resource from a string.

New `Resource` transformation funcs:

* `resources.ToCSS`: Compile `SCSS` or `SASS` into `CSS`.
* `resources.PostCSS`: Process your CSS with PostCSS. Config file support (project or theme or passed as an option).
* `resources.Minify`: Currently supports `css`, `js`, `json`, `html`, `svg`, `xml`.
* `resources.Fingerprint`: Creates a fingerprinted version of the given Resource with Subresource Integrity..
* `resources.Concat`: Concatenates a list of Resource objects. Think of this as a poor man's bundler.
* `resources.ExecuteAsTemplate`: Parses and executes the given Resource and data context (e.g. .Site) as a Go template.

Fixes #4381
Fixes #4903
Fixes #4858
2018-07-06 11:46:12 +02:00
Bjørn Erik Pedersen 159bed34c3
parser: Add some context to front matter parse error
Fixes #4638
2018-04-17 10:20:38 +02:00
Bjørn Erik Pedersen 0816a97a46
parser: Add WARNING for integer YAML keys
```bash
benchmark                                               old ns/op     new ns/op     delta
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4     3053          2015          -34.00%
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        5.23          5.18          -0.96%
BenchmarkStringifyMapKeysIntegers-4                     2320          5177          +123.15%

benchmark                                               old allocs     new allocs     delta
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4     6              6              +0.00%
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        0              0              +0.00%
BenchmarkStringifyMapKeysIntegers-4                     6              14             +133.33%

benchmark                                               old bytes     new bytes     delta
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4     1008          1008          +0.00%
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        0             0             +0.00%
BenchmarkStringifyMapKeysIntegers-4                     1008          1776          +76.19%
```
Closes #4393
2018-02-12 19:16:12 +01:00
Bjørn Erik Pedersen 10a917dfdc
parser: Tune stringifyMapKeys
```bash
benchmark                                               old ns/op     new ns/op     delta
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4     3269          3053          -6.61%
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        4.79          5.23          +9.19%
BenchmarkStringifyMapKeysIntegers-4                     2707          2320          -14.30%

benchmark                                               old allocs     new allocs     delta
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4     16             6              -62.50%
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        0              0              +0.00%
BenchmarkStringifyMapKeysIntegers-4                     16             6              -62.50%

benchmark                                               old bytes     new bytes     delta
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4     1080          1008          -6.67%
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        0             0             +0.00%
BenchmarkStringifyMapKeysIntegers-4                     1080          1008          -6.67%
```
2018-02-12 19:14:02 +01:00
Bjørn Erik Pedersen d4beef0d2b
parser: Rename stringifyYAMLMapKeys to stringifyMapKeys 2018-02-12 18:17:25 +01:00
Bjørn Erik Pedersen 51213e0be1
parser: Add benchmarks for stringifyYAMLMapKeys
```bash
BenchmarkStringifyMapKeysStringsOnlyInterfaceMaps-4       500000              3269 ns/op            1080 B/op         16 allocs/op
BenchmarkStringifyMapKeysStringsOnlyStringMaps-4        300000000                4.79 ns/op            0 B/op          0 allocs/op
BenchmarkStringifyMapKeysIntegers-4                       500000              2707 ns/op            1080 B/op         16 allocs/op
```
2018-02-12 18:16:48 +01:00
Vas Sudanagunta 1fa2417777 Add support for YAML array data files
* Unmarshaled YAML arrays indistinguishable from JSON arrays.
* Fixes #3890
2018-02-12 17:14:40 +01:00
Dawid Gaweł 16a5c74519 parser: Fix YAML maps key type
Recurse through result of yaml package parsing and change all
maps from map[interface{}]interface{} to map[string]interface{}
making them jsonable and sortable.

Fixes #2441, #4083
2018-02-09 10:27:17 +01:00
Vas Sudanagunta 4402c07775 Fix JSON array-based data file handling regression
This bug was introduced in Hugo 0.35.

Fixes #4361
2018-02-02 09:14:37 +01:00
Vas Sudanagunta 91bb774ae4 Support pages without front matter
* Page without front matter now treated same as a page with empty front matter.
* Test cases added to cover this and repro issue #4320.
* Type safety of front matter code improved.

Fixes #4320
2018-01-26 09:17:27 +01:00
Bjørn Erik Pedersen 7f82b41a24 parser: Final (!) fix for issue with escaped JSON front matter
Fixes #3682
2017-07-08 18:43:36 +02:00
Bjørn Erik Pedersen 84db6c74a0 parser: Fix issue with escaped JSON front matter
Fixes #3682
2017-07-08 10:56:56 +02:00
Bjørn Erik Pedersen e10e51a008 parser: Fix handling of JSON front matter with escaped quotes
Fixes #3661
2017-07-03 09:01:54 +02:00
Bjørn Erik Pedersen 3183b9a29d parser: Fix handling of quoted brackets in JSON front matter
Also remove a broken JSON test.

Fixes #3511
2017-06-19 16:45:52 +02:00
Bjørn Erik Pedersen 422057f607 create: Use archetype template as-is as a Go template
This commit removes the fragile front matter decoding, and takes the provided archetype file as-is and processes it as a template.

This also means that we no longer will attempt to fill in default values for `title` and `date`.

The upside is that it is now easy to create these values in a dynamic way:

```toml
+++
title = {{ .BaseFileName | title }}
date = {{ .Date }}
draft = true
+++
```

You can currently use all of Hugo's template funcs, but the data context is currently very shallow:

* `.Type` gives the archetype kind provided
* `.Name` gives the target file name without extension.
* `.Path` gives the target file name
* `.Date` gives the current time as RFC3339 formatted string

The above  will probably be extended in #1629.

Fixes #452
Updates #1629
2017-06-18 19:06:28 +02:00
Bjørn Erik Pedersen 2c2ce33a39 parser: Add horizontal YAML tags to benchmark 2017-06-03 12:35:28 +02:00
Bjørn Erik Pedersen 0907a5c1c2 all: Temporarily revert to BurntSushi for TOML front matter handling
We still have go-toml as a transitive dependency, and it is the way to go eventually, but we care about speed, so let us wait that one out.

Note that the issue this fixes is about taxonomies, but I guess this is a general issue for sites with many pages that uses TOML as front matter.

```
benchmark                              old ns/op     new ns/op     delta
BenchmarkFrontmatterTags/TOML:1-4      23206         8543          -63.19%
BenchmarkFrontmatterTags/TOML:11-4     80117         18495         -76.92%
BenchmarkFrontmatterTags/TOML:21-4     140676        28727         -79.58%

benchmark                              old allocs     new allocs     delta
BenchmarkFrontmatterTags/TOML:1-4      173            60             -65.32%
BenchmarkFrontmatterTags/TOML:11-4     625            138            -77.92%
BenchmarkFrontmatterTags/TOML:21-4     1106           210            -81.01%

benchmark                              old bytes     new bytes     delta
BenchmarkFrontmatterTags/TOML:1-4      9231          2912          -68.45%
BenchmarkFrontmatterTags/TOML:11-4     19808         5184          -73.83%
BenchmarkFrontmatterTags/TOML:21-4     31200         7536          -75.85%
```

See #3541
Updates #3464
2017-06-03 09:22:57 +02:00
Bjørn Erik Pedersen 3d9c4f513b parser: Add BenchmarkFrontmatterTags
The list handling is surprisingly expensive:

```
▶ go test -run="NONE" -bench="BenchmarkFrontmatterTags" -test.benchmem=true ./parser | prettybench
PASS
benchmark                               iter         time/iter   bytes alloc           allocs
---------                               ----         ---------   -----------           ------
BenchmarkFrontmatterTags/JSON:1-4    1000000     2039.00 ns/op      912 B/op     20 allocs/op
BenchmarkFrontmatterTags/JSON:11-4    300000     5202.00 ns/op     1640 B/op     44 allocs/op
BenchmarkFrontmatterTags/JSON:21-4    200000     7993.00 ns/op     2392 B/op     65 allocs/op
BenchmarkFrontmatterTags/YAML:1-4     200000     9359.00 ns/op     5928 B/op     66 allocs/op
BenchmarkFrontmatterTags/YAML:11-4    100000    21218.00 ns/op     8408 B/op    140 allocs/op
BenchmarkFrontmatterTags/YAML:21-4     50000    32852.00 ns/op    10920 B/op    211 allocs/op
BenchmarkFrontmatterTags/TOML:1-4     100000    21505.00 ns/op     9231 B/op    173 allocs/op
BenchmarkFrontmatterTags/TOML:11-4     20000    82919.00 ns/op    19808 B/op    625 allocs/op
BenchmarkFrontmatterTags/TOML:21-4     10000   141847.00 ns/op    31200 B/op   1106 allocs/op
ok      github.com/spf13/hugo/parser    17.890s
```

See #3464
2017-06-03 08:53:56 +02:00
Albert Nigmatzianov d3cd68e656 parser: Improve TOML frontmatter parser performance
Difference between toml.Load(string(datum)) and
toml.LoadReader(bytes.NewReader(datum)):
benchmark           old ns/op     new ns/op     delta
BenchmarkLoad-4     82068         78489         -4.36%

benchmark           old allocs     new allocs     delta
BenchmarkLoad-4     494            493            -0.20%

benchmark           old bytes     new bytes     delta
BenchmarkLoad-4     17009         16913         -0.56%
2017-03-20 09:02:35 +01:00
Bjørn Erik Pedersen d8923eb676 Update to new go-toml API
Closes #3142
2017-03-15 08:04:52 +01:00
crasm ede452d34e parser: Accept JSON frontmatter without leading "{\n"
Accept JSON frontmatter without leading "{\n" so that one line
frontmatters such as `{"param":"paramvalue"}` no longer silently render
empty html.
2017-03-13 17:19:03 +01:00
Cameron Moore f039e3be9e parser: Refactor frontmatter parser and add tests
Lots of cleanups here:

- Refactor InterfaceToConfig and InterfaceToFrontMatter to use io.Writer.
- Simplify InterfaceToFrontMatter by wrapping InterfaceToConfig.
- Export FrontmatterType since we return it in DetectFrontMatter.
- Refactor removeTOMLIdentifier to avoid blindly replacing "+++".
- Update HandleJSONMetaData to return an empty map on nil input.
- Updates vendored goorgeous package and test for org-mode frontmatter.
- Add tests and godoc comments.

Coverage for parser package increased from 45.2% to 85.2%.
2017-03-11 17:52:25 +01:00
Chase Adams 86e8dd62f0 all: Add org-mode support
Fixes #1483 
See #936
2017-02-21 08:46:03 +01:00
bogem dec1706ae0 commands, hugolib, parser, tpl: Use errors.New instead of fmt.Errorf 2016-11-22 23:43:55 +01:00
bogem 4d1cb50003 parser: Use strings.Contains instead of strings.Index
Closes #2400
2016-09-11 12:23:15 +02:00
bogem 798bf60c56 parser: Simplify err conditions 2016-09-11 12:23:09 +02:00
Bjørn Erik Pedersen 20c4311df4 Switch to a more up to date TOML library
Fixes #2089
2016-08-20 20:33:06 +01:00
Henrique Dias af34613f75 Update frontmatter.go 2016-07-13 23:53:06 +02:00
Henrique Dias 16b71bbbb4 Export "detectFrontMatter" to fix caddy-hugo
Closes #2108
2016-04-26 22:28:47 +02:00
Mattias Wadman 5d50c46482 Chomp Unicode BOM if present
Useful if using or sharing files with users that use editors that
append a unicode byte order marker header (like Windows notepad).

This will still assume files are UTF-8 encoded.

Closes #2075
2016-04-14 10:46:27 +02:00
Bjørn Erik Pedersen b71f391cd7 parser: Remove unused vars 2016-03-24 21:34:34 +01:00
Bjørn Erik Pedersen c20dee9d7f parser: Make the constant vars ... constants 2016-03-23 15:05:32 +01:00
Bjørn Erik Pedersen bf76e74432 parser: Fix ALL_CAPS var names in test 2016-03-23 15:02:00 +01:00
Bjørn Erik Pedersen 1cb7ed6ac7 parser: Spring code cleaning 2016-03-23 14:51:16 +01:00
Bjørn Erik Pedersen 3a82ae7114 parser: Unexport some internals 2016-03-14 17:52:11 +01:00
Bjørn Erik Pedersen df3f2af0a8 parser: Remove unused code 2016-03-14 15:42:59 +01:00
Derek Perkins bac1ba4655 Fix spelling and go vet errors 2016-02-13 19:08:48 +08:00