The other named errors - UnmarshalTypeError, etc - in this package do
the same, so we should prepend the package prefix to error messages
for consistency.
Add a note to the release docs in case this is interpreted as
a breaking change.
Fixes#36221.
Change-Id: Ie24b532bbf9812e108c259fa377e2a6b64319ed4
Reviewed-on: https://go-review.googlesource.com/c/go/+/263619
Run-TryBot: Kevin Burke <kev@inburke.com>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Kevin Burke <kev@inburke.com>
Trust: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
The old ioutil references are still valid, but update our code
to reflect best practices and get used to the new locations.
Code compiled with the bootstrap toolchain
(cmd/asm, cmd/dist, cmd/compile, debug/elf)
must remain Go 1.4-compatible and is excluded.
Also excluded vendored code.
For #41190.
Change-Id: I6d86f2bf7bc37a9d904b6cee3fe0c7af6d94d5b1
Reviewed-on: https://go-review.googlesource.com/c/go/+/263142
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Now reports an error if cyclic maps and slices are to be encoded
instead of an infinite recursion. This case wasn't handled in CL 187920.
Fixes#40745.
Change-Id: Ia34b014ecbb71fd2663bb065ba5355a307dbcc15
GitHub-Last-Rev: 6f874944f4
GitHub-Pull-Request: golang/go#40756
Reviewed-on: https://go-review.googlesource.com/c/go/+/248358
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Trust: Daniel Martí <mvdan@mvdan.cc>
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Go Bot <gobot@golang.org>
Allow ';' as a valid character for json field keys and struct tags.
Fixes#39189
Change-Id: I4b602a1b0674ff028db07623682f0d1e8e9fd6c9
Reviewed-on: https://go-review.googlesource.com/c/go/+/234818
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Giovanni Bajo <rasky@develer.com>
Trust: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
This reverts CL 253037.
Reason for revert: The recommended way to check for a type of error is errors.As. API changes should also start with a proposal.
Change-Id: I62896717aa47ed491c2c4775d2b05d80e5e9cde3
Reviewed-on: https://go-review.googlesource.com/c/go/+/254837
Trust: Damien Neil <dneil@google.com>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
This reverts CL 254537.
Reason for revert: Reason for revert: The recommended way to check for a type of error is errors.As. API changes should also start with a proposal.
Change-Id: I07c37428575e99c80b17525833a61831d10963bb
Reviewed-on: https://go-review.googlesource.com/c/go/+/254857
Trust: Damien Neil <dneil@google.com>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Allows users to check:
errors.Is(err, &UnmarshalTypeError{})
errors.Is(err, &UnmarshalFieldError{})
errors.Is(err, &InvalidUnmarshalError{})
errors.Is(err, &UnsupportedValueError{})
errors.Is(err, &MarshalerError{})
which is the recommended way of checking for kinds of errors.
SyntaxError.Is was implemented in CL 253037.
As and Unwrap relevant methods will be added in future CLs.
Change-Id: I1f8a503b8fdc0f3afdfe9669a91f3af8d960e028
GitHub-Last-Rev: 930cda5384
GitHub-Pull-Request: golang/go#41360
Reviewed-on: https://go-review.googlesource.com/c/go/+/254537
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
Allows users to check:
errors.Is(err, &json.SyntaxError{})
which is the recommended way of checking for kinds of errors.
Change-Id: I20dc805f20212765e9936a82d9cb7822e73ec4ef
GitHub-Last-Rev: e2627ccf8e
GitHub-Pull-Request: golang/go#41210
Reviewed-on: https://go-review.googlesource.com/c/go/+/253037
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This reverts https://golang.org/cl/191783.
Reason for revert: Broke too many programs which depended on the previous
behavior, even when it was the opposite of what the documentation said.
We can attempt to fix the original issue again for 1.16, while keeping
those programs in mind.
Fixes#39427.
Change-Id: I7a7f24b2a594c597ef625aeff04fff29aaa88fc6
Reviewed-on: https://go-review.googlesource.com/c/go/+/240657
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This reverts golang.org/cl/190659 and golang.org/cl/226218, minus the
regression tests in the latter.
The original work happened in golang.org/cl/151157, which was reverted
in golang.org/cl/190909 due to a crash found by fuzzing.
We tried a second time in golang.org/cl/190659, which shipped with Go
1.14. A bug was found, where strings would be mangled in certain edge
cases. The fix for that was golang.org/cl/226218, which was backported
into Go 1.14.4.
Unfortunately, a second regression was just reported in #39555, which is
a similar case of strings getting mangled when decoding under certain
conditions. It would be possible to come up with another small patch to
fix that edge case, but instead, let's just revert the entire
optimization, as it has proved to do more harm than good. Moreover, it's
hard to argue or prove that there will be no more such regressions.
However, all the work wasn't for nothing. First, we learned that the way
the decoder unquotes tokenized strings isn't simple; initially, we had
wrongly assumed that each string was unquoted exactly once and in order.
Second, we have gained a number of regression tests which will be useful
to prevent the same mistakes in the future, including the test cases we
add in this CL.
Fixes#39555.
Change-Id: I66a6919c2dd6d9789232482ba6cf3814eaa70f61
Reviewed-on: https://go-review.googlesource.com/c/go/+/237838
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
This reverts golang.org/cl/179337.
Reason for revert: broke a few too many reasonably valid Go programs.
The previous behavior was perhaps less consistent, but the docs were
never very clear about when the decoder merges with existing values,
versus replacing existing values altogether.
Fixes#39149.
Change-Id: I1c1d857709b8398969fe421aa962f6b62f91763a
Reviewed-on: https://go-review.googlesource.com/c/go/+/234559
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
When we decode into a struct, each input key-value may be decoded into
one of the struct's fields. Particularly, existing data isn't dropped,
so that some sub-fields can be decoded into without zeroing all other
data.
However, decoding into a map behaved in the opposite way. Whenever a
key-value was decoded, it completely replaced the previous map element.
If the map contained any non-zero data in that key, it's dropped.
Instead, try to reuse the existing element value if possible. If the map
element type is a pointer, and the value is non-nil, we can decode
directly into it. If it's not a pointer, make a copy and decode into
that copy, as map element values aren't addressable.
This means we have to parse and convert the map element key before the
value, to be able to obtain the existing element value. This is fine,
though. Moreover, reporting errors on the key before the value follows
the input order more closely.
Finally, add a test to explore the four combinations, involving pointer
and non-pointer, and non-zero and zero values. A table-driven test
wasn't used, as each case required different checks, such as checking
that the non-nil pointer case doesn't end up with a different pointer.
Fixes#31924.
Change-Id: I5ca40c9963a98aaf92f26f0b35843c021028dfca
Reviewed-on: https://go-review.googlesource.com/c/go/+/179337
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The added comment contains some context. The original optimization
assumed that each call to unquoteBytes (or unquote) followed its
corresponding call to rescanLiteral. Otherwise, unquoting a literal
might use d.safeUnquote from another re-scanned literal.
Unfortunately, this assumption is wrong. When decoding {"foo": "bar"}
into a map[T]string where T implements TextUnmarshaler, the sequence of
calls would be as follows:
1) rescanLiteral "foo"
2) unquoteBytes "foo"
3) rescanLiteral "bar"
4) unquoteBytes "foo" (for UnmarshalText)
5) unquoteBytes "bar"
Note that the call to UnmarshalText happens in literalStore, which
repeats the work to unquote the input string literal. But, since that
happens after we've re-scanned "bar", we're using the wrong safeUnquote
field value.
In the added test case, the second string had a non-zero number of safe
bytes, and the first string had none since it was all non-ASCII. Thus,
"safely" unquoting a number of the first string's bytes could cut a rune
in half, and thus mangle the runes.
A rather simple fix, without a full revert, is to only allow one use of
safeUnquote per call to unquoteBytes. Each call to rescanLiteral when
we have a string is soon followed by a call to unquoteBytes, so it's no
longer possible for us to use the wrong index.
Also add a test case from #38126, which is the same underlying bug, but
affecting the ",string" option.
Before the fix, the test would fail, just like in the original two issues:
--- FAIL: TestUnmarshalRescanLiteralMangledUnquote (0.00s)
decode_test.go:2443: Key "开源" does not exist in map: map[开���:12345开源]
decode_test.go:2458: Unmarshal unexpected error: json: invalid use of ,string struct tag, trying to unmarshal "\"aaa\tbbb\"" into string
Fixes#38105.
For #38126.
Change-Id: I761e54924e9a971a4f9eaa70bbf72014bb1476e6
Reviewed-on: https://go-review.googlesource.com/c/go/+/226218
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
golang.org/cl/193604 fixed one bug when one encodes a string with the
",string" option: if SetEscapeHTML(false) is used, we should not be
using HTML escaping for the inner string encoding. The CL correctly
fixed that.
The CL also tried to speed up this edge case. By avoiding an entire new
call to Marshal, the new Issue34127 benchmark reduced its time/op by
45%, and lowered the allocs/op from 3 to 2.
However, that last optimization wasn't correct:
Since Go 1.2 every string can be marshaled to JSON without error
even if it contains invalid UTF-8 byte sequences. Therefore
there is no need to use Marshal again for the only reason of
enclosing the string in double quotes.
JSON string encoding isn't just about adding quotes and taking care of
invalid UTF-8. We also need to escape some characters, like tabs and
newlines.
The new code failed to do that. The bug resulted in the added test case
failing to roundtrip properly; before our fix here, we'd see an error:
invalid use of ,string struct tag, trying to unmarshal "\"\b\f\n\r\t\"\\\"" into string
If you pay close attention, you'll notice that the special characters
like tab and newline are only encoded once, not twice. When decoding
with the ",string" option, the outer string decode works, but the inner
string decode fails, as we are now decoding a JSON string with unescaped
special characters.
The fix we apply here isn't to go back to Marshal, as that would
re-introduce the bug with SetEscapeHTML(false). Instead, we can use a
new encode state from the pool - it results in minimal performance
impact, and even reduces allocs/op further. The performance impact seems
fair, given that we need to check the entire string for characters that
need to be escaped.
name old time/op new time/op delta
Issue34127-8 89.7ns ± 2% 100.8ns ± 1% +12.27% (p=0.000 n=8+8)
name old alloc/op new alloc/op delta
Issue34127-8 40.0B ± 0% 32.0B ± 0% -20.00% (p=0.000 n=8+8)
name old allocs/op new allocs/op delta
Issue34127-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=8+8)
Instead of adding another standalone test, we convert an existing
"string tag" test to be table-based, and add another test case there.
One test case from the original CL also had to be amended, due to the
same problem - when escaping '<' due to SetEscapeHTML(true), we need to
end up with double escaping, since we're using ",string".
Fixes#38173.
Change-Id: I2b0df9e4f1d3452fff74fe910e189c930dde4b5b
Reviewed-on: https://go-review.googlesource.com/c/go/+/226498
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The previous behavior directly contradicted the docs that have been in
place for years:
To unmarshal a JSON array into a slice, Unmarshal resets the
slice length to zero and then appends each element to the slice.
We could use reflect.New to create a new element and reflect.Append to
then append it to the destination slice, but benchmarks have shown that
reflect.Append is very slow compared to the code that manually grows a
slice in this file.
Instead, if we're decoding into an element that came from the original
backing array, zero it before decoding into it. We're going to be using
the CodeDecoder benchmark, as it has a slice of struct pointers that's
decoded very often.
Note that we still reuse existing values from arrays being decoded into,
as the documentation agrees with the existing implementation in that
case:
To unmarshal a JSON array into a Go array, Unmarshal decodes
JSON array elements into corresponding Go array elements.
The numbers with the benchmark as-is might seem catastrophic, but that's
only because the benchmark is decoding into the same variable over and
over again. Since the old decoder was happy to reuse slice elements, it
would save a lot of allocations by not having to zero and re-allocate
said elements:
name old time/op new time/op delta
CodeDecoder-8 10.4ms ± 1% 10.9ms ± 1% +4.41% (p=0.000 n=10+10)
name old speed new speed delta
CodeDecoder-8 186MB/s ± 1% 178MB/s ± 1% -4.23% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
CodeDecoder-8 2.19MB ± 0% 3.59MB ± 0% +64.09% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
CodeDecoder-8 76.8k ± 0% 92.7k ± 0% +20.71% (p=0.000 n=10+10)
We can prove this by moving 'var r codeResponse' into the loop, so that
the benchmark no longer reuses the destination pointer. And sure enough,
we no longer see the slow-down caused by the extra allocations:
name old time/op new time/op delta
CodeDecoder-8 10.9ms ± 0% 10.9ms ± 1% -0.37% (p=0.043 n=10+10)
name old speed new speed delta
CodeDecoder-8 177MB/s ± 0% 178MB/s ± 1% +0.37% (p=0.041 n=10+10)
name old alloc/op new alloc/op delta
CodeDecoder-8 3.59MB ± 0% 3.59MB ± 0% ~ (p=0.780 n=10+10)
name old allocs/op new allocs/op delta
CodeDecoder-8 92.7k ± 0% 92.7k ± 0% ~ (all equal)
I believe that it's useful to leave the benchmarks as they are now,
because the decoder does reuse memory in some cases. For example,
existing map elements are reused. However, subtle changes like this one
need to be benchmarked carefully.
Finally, add a couple of tests involving both a slice and an array of
structs.
Fixes#21092.
Change-Id: I8b1194f25e723a31abd146fbfe9428ac10c1389d
Reviewed-on: https://go-review.googlesource.com/c/go/+/191783
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Limit the maximum nesting depth when parsing to protect against stack
overflow, permitted by https://tools.ietf.org/html/rfc7159#section-9
A nesting depth limit of 10,000 was chosen to be a conservative
balance between avoiding stack overflow and avoiding impacting
legitimate JSON documents.
10,000 is less than 1% of the experimental stack depth limit
with the default stack size:
* On 64-bit systems, the default stack limit is 1GB,
which allows ~2,800,000 frames of recursive parsing
* On 32-bit systems, the default stack limit is 250MB,
which allows ~1,100,000 frames of recursive parsing
Fixes#31789
Change-Id: I4f5a90e89dcb4ab1a957ad9d02e1fa0efafaccf6
Reviewed-on: https://go-review.googlesource.com/c/go/+/199837
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
This updates the link to a newer image.
Change-Id: Ibdfe8c57d9217a325bcfde98cb6f952ca63d588a
GitHub-Last-Rev: f5970ba395
GitHub-Pull-Request: golang/go#36938
Reviewed-on: https://go-review.googlesource.com/c/go/+/217297
Reviewed-by: Toshihiro Shiino <shiino.toshihiro@gmail.com>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Otherwise we'd panic with a stack overflow.
Most programs are in control of the data being encoded and can ensure
there are no cycles, but sometimes it's not that simple. For example,
running a user's html template with script tags can easily result in
crashes if the user can find a pointer cycle.
Adding the checks via a map to every ptrEncoder.encode call slowed down
the benchmarks below by a noticeable 13%. Instead, only start doing the
relatively expensive pointer cycle checks if we're many levels of
pointers deep in an encode state.
A threshold of 1000 is small enough to capture pointer cycles before
they're a problem (the goroutine stack limit is currently 1GB, and I
needed close to a million levels to reach it). Yet it's large enough
that reasonable uses of the json encoder only see a tiny 1% slow-down
due to the added ptrLevel field and check.
name old time/op new time/op delta
CodeEncoder-8 2.34ms ± 1% 2.37ms ± 0% +1.05% (p=0.000 n=10+10)
CodeMarshal-8 2.42ms ± 1% 2.44ms ± 0% +1.10% (p=0.000 n=10+10)
name old speed new speed delta
CodeEncoder-8 829MB/s ± 1% 820MB/s ± 0% -1.04% (p=0.000 n=10+10)
CodeMarshal-8 803MB/s ± 1% 795MB/s ± 0% -1.09% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
CodeEncoder-8 43.1kB ± 8% 42.5kB ±10% ~ (p=0.989 n=10+10)
CodeMarshal-8 1.99MB ± 0% 1.99MB ± 0% ~ (p=0.254 n=9+6)
name old allocs/op new allocs/op delta
CodeEncoder-8 0.00 0.00 ~ (all equal)
CodeMarshal-8 1.00 ± 0% 1.00 ± 0% ~ (all equal)
Finally, add a few tests to ensure that the code handles the edge cases
properly.
Fixes#10769.
Change-Id: I73d48e0cf6ea140127ea031f2dbae6e6a55e58b8
Reviewed-on: https://go-review.googlesource.com/c/go/+/187920
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bjørn Erik Pedersen <bjorn.erik.pedersen@gmail.com>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
This is a re-submission of CL 151157, since it was reverted in CL 190909
due to an introduced crash found by a fuzzer. The revert CL included
regression tests, while this CL includes a fixed version of the original
change.
In particular, what we forgot in the original optimization was that we
still need the length and trailing quote checks at the beginning of
unquoteBytes. Without those, we could end up in a crash later on.
We can work out how many bytes can be unquoted trivially in
rescanLiteral, which already iterates over a string's bytes.
Removing the extra loop in unquoteBytes simplifies the function and
speeds it up, especially when decoding simple strings, which are common.
While at it, we can remove the check that s[0]=='"', since all call
sites already meet that condition.
name old time/op new time/op delta
CodeDecoder-8 10.6ms ± 2% 10.5ms ± 1% -1.01% (p=0.004 n=20+10)
name old speed new speed delta
CodeDecoder-8 183MB/s ± 2% 185MB/s ± 1% +1.02% (p=0.003 n=20+10)
Updates #28923.
Change-Id: I8c6b13302bcd86a364bc998d72451332c0809cde
Reviewed-on: https://go-review.googlesource.com/c/go/+/190659
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Peter Weinberger <pjw@google.com>
If we marshal a non-pointer struct field whose type implements Marshaler with
a non-pointer receiver, then we avoid an allocation if we take the address of
the field before casting it to an interface.
name old time/op new time/op delta
EncodeMarshaler-8 104ns ± 1% 92ns ± 2% -11.72% (p=0.001 n=7+7)
name old alloc/op new alloc/op delta
EncodeMarshaler-8 36.0B ± 0% 4.0B ± 0% -88.89% (p=0.000 n=8+8)
name old allocs/op new allocs/op delta
EncodeMarshaler-8 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.000 n=8+8)
Test coverage already looks good enough for this change. TestRefValMarshal
already covers all possible combinations of value & pointer receivers on
value and pointer struct fields.
Change-Id: I6fc7f72396396d98f9a90c3c86e813690f41c099
Reviewed-on: https://go-review.googlesource.com/c/go/+/203608
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This change improves performance of Compact by using a sync.Pool to allow re-use
of a scanner. This also has the side-effect of removing an allocation for each
field that implements Marshaler when marshalling JSON.
name old time/op new time/op delta
EncodeMarshaler-8 118ns ± 2% 104ns ± 1% -12.21% (p=0.001 n=7+7)
name old alloc/op new alloc/op delta
EncodeMarshaler-8 100B ± 0% 36B ± 0% -64.00% (p=0.000 n=8+8)
name old allocs/op new allocs/op delta
EncodeMarshaler-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=8+8)
Change-Id: Ic70c61a0a6354823da5220f5aad04b94c054f233
Reviewed-on: https://go-review.googlesource.com/c/go/+/200864
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
1. Change mapencode.encode to use fmt.Error rather than MarshalerError.
MarshalerError refer to MarshalJSON, but mapencode.encode does not use that.
2. Add sourceFunc field to MarshalerError to record the name of the function
that creates the error, so that the Error method can report it correctly.
Fixes#29753
Change-Id: I186c2fac8470ae2f9e300501de3730face642230
Reviewed-on: https://go-review.googlesource.com/c/go/+/184119
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This makes Decoder.offset public while renaming it to
Decoder.InputOffset to match encoding/xml Decoder API
Code changes made by Adam Stankiewicz [sheerun@sher.pl]
Fixes#29688
Change-Id: I86dbfd2b2da80160846e92bfa580c53d8d45e2db
Reviewed-on: https://go-review.googlesource.com/c/go/+/200677
Run-TryBot: Johan Brandhorst <johan.brandhorst@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
When unmarshaling to a map, the map's key type must either be a string,
an integer, or implement encoding.TextUnmarshaler. But for a user
defined type, reflect.Kind will not distinguish between the static type
and the underlying type. In:
var x MyString = "x"
t := reflect.TypeOf(x)
println(t.Kind() == reflect.String)
the Kind of x is still reflect.String, even though the static type of x
is MyString.
Moreover, checking for the map's key type is a string occurs first, so
even if the map key type MyString implements encoding.TextUnmarshaler,
it will be ignored.
To fix the bug, check for encoding.TextUnmarshaler first.
Fixes#34437
Change-Id: I780e0b084575e1dddfbb433fe03857adf71d05fb
Reviewed-on: https://go-review.googlesource.com/c/go/+/200237
Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Compact has been inconsistently escaping only some problematic characters
(U+2028 and U+2029), but not others (<, > and &). This change addresses
this inconsistency by removing the escaping of U+2028 and U+2029.
Callers who need to escape the output of Compact should use HTMLEscape
which escapes <, >, &, U+2028 and U+2029.
Fixes#34070Fixes#30357
Updates #5836
Change-Id: Icfce7691d2b8b1d9b05ba7b64d2d1e4f3b67871b
GitHub-Last-Rev: 38859fe3e2
GitHub-Pull-Request: golang/go#34804
Reviewed-on: https://go-review.googlesource.com/c/go/+/200217
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Unmarshaling a string into a json.Number should first check that the string is a valid Number.
If not, we should fail without decoding it.
Fixes#14702
Change-Id: I286178e93df74ad63c0a852c3f3489577072cf47
GitHub-Last-Rev: fe69bb68ee
GitHub-Pull-Request: golang/go#34272
Reviewed-on: https://go-review.googlesource.com/c/go/+/195045
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Add quotes when marshaling a json.Number with the string option
set via a struct tag. This ensures that the resulting json
can be unmarshaled into the source struct without error.
Fixes#34268
Change-Id: Ide167d9dec77019554870b5957b37dc258119d81
GitHub-Last-Rev: dde81b7120
GitHub-Pull-Request: golang/go#34269
Reviewed-on: https://go-review.googlesource.com/c/go/+/195043
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Since Go 1.2 every string can be marshaled to JSON without error even if it
contains invalid UTF-8 byte sequences. Therefore there is no need to use
Marshal again for the only reason of enclosing the string in double quotes.
Not using Marshal here also removes the error check as there has not been a
way for Marshal to fail anyway.
name old time/op new time/op delta
Issue34127-4 360ns ± 3% 200ns ± 3% -44.56% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
Issue34127-4 56.0B ± 0% 40.0B ± 0% -28.57% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
Issue34127-4 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.008 n=5+5)
Fixes#34154
Change-Id: Ib60dc11980f9b20d8bef2982de7168943d632263
GitHub-Last-Rev: 9b0ac1d4c5
GitHub-Pull-Request: golang/go#34127
Reviewed-on: https://go-review.googlesource.com/c/go/+/193604
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reset is already performed when retrieving from pool
Change-Id: Ia810dd18d3e55a1565a5ad435a00d1e46724576c
GitHub-Last-Rev: d9df74a4ae
GitHub-Pull-Request: golang/go#34195
Reviewed-on: https://go-review.googlesource.com/c/go/+/194338
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
The indirect method checked the type of the child when indirecting a
pointer. If the current value is a pointer and we are decoding null, we
can skip this entirely and return early, avoiding the whole descent.
Fixes#31776
Change-Id: Ib8b2a2357572c41f56fceac59b5a858980f3f65e
Reviewed-on: https://go-review.googlesource.com/c/go/+/174699
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
This is a documentation-only change
Fixes#33298
Change-Id: I816058a872b57dc868dff11887214d9de92d9342
Reviewed-on: https://go-review.googlesource.com/c/go/+/188821
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Some were never used, and some haven't been used for years.
One exception is net/http's readerAndCloser, which was only used in a
test. Move it to a test file.
While at it, remove a check in regexp that could never fire; the field
is an uint32, so it can never be negative.
Change-Id: Ia2200f6afa106bae4034045ea8233b452f38747b
Reviewed-on: https://go-review.googlesource.com/c/go/+/192621
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
scanEnd is delayed one byte so we decrement
the scanner bytes count by 1 to ensure that
this value is correct in the next call of Decode.
Fixes#32399
Change-Id: I8c8698e7f95bbcf0373aceaa05319819eae9d86f
GitHub-Last-Rev: 0ac25d8de2
GitHub-Pull-Request: golang/go#32598
Reviewed-on: https://go-review.googlesource.com/c/go/+/182117
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This partly reverts CL 173417 as it incorrectly documented that Compact
performed HTML escaping and the output was safe to embed inside HTML
<script> tags. This has never been true.
Although Compact does escape U+2028 and U+2029, it doesn't escape <, >
or &. Compact is thus only performing a subset of HTML escaping and it's
output is not safe to embed inside HTML <script> tags.
A more complete fix would be for Compact to either never perform any
HTML escaping, as it was prior to CL 10883045, or to actually perform
the same HTML escaping as HTMLEscape. Neither change is likely safe
enough for go1.13.
Updates #30357
Change-Id: I912f0fe9611097d988048b28228c4a5b985080ba
GitHub-Last-Rev: aebababc92
GitHub-Pull-Request: golang/go#33427
Reviewed-on: https://go-review.googlesource.com/c/go/+/188717
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This change adds a a check in the encodeWithString.resolve method
to ensure that a reflect.Value with kind Ptr is not nil before
the type assertion to TextMarshaler.
If the value is nil, the method returns a nil error, and the map key
encodes to an empty string.
Fixes#33675
Change-Id: I0a04cf690ae67006f6a9c5f8cbb4cc99d236bca8
GitHub-Last-Rev: 6c987c9084
GitHub-Pull-Request: golang/go#33700
Reviewed-on: https://go-review.googlesource.com/c/go/+/190697
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
The decoder called this function to check numbers being decoded into a
json.Number. However, these can't be quoted as strings, so the tokenizer
has already verified they are valid JSON numbers.
Verified this by adding a test with such an input. As expected, it
produces a syntax error, not the fmt.Errorf - that line could never
execute.
Since the only remaining non-test caller of isvalidnumber is in
encode.go, move the function there.
This change should slightly reduce the amount of work when decoding into
json.Number, though that isn't very common nor part of any current
benchmarks.
Change-Id: I67a1723deb3d18d5b542d6dd35f3ae56a43f23eb
Reviewed-on: https://go-review.googlesource.com/c/go/+/184817
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Because TestUnmarshal actually allocates a new value to decode into
using ptr's pointer type, any existing data is thrown away. This was
harmless in alomst all of the test cases, minus the "overwriting of
data" ones added in 2015 in CL 12209.
I spotted that nothing covered decoding a JSON array with few elements
into a slice which already had many elements. I initially assumed that
the code was buggy or that some code could be removed, when in fact
there simply wasn't any code covering the edge case.
Move those two tests to TestPrefilled, which already served a very
similar purpose. Remove the map case, as TestPrefilled already has
plenty of prefilled map cases. Moreover, we no longer reset an entire
map when decoding, as per the godoc:
To unmarshal a JSON object into a map, Unmarshal first
establishes a map to use. If the map is nil, Unmarshal allocates
a new map. Otherwise Unmarshal reuses the existing map, keeping
existing entries.
Finally, to ensure that ptr is used correctly in the future, make
TestUnmarshal error if it's anything other than a pointer to a zero
value. That is, the only correct use should be new(type). Don't rename
the ptr field, as that would be extremely noisy and cause unwanted merge
conflicts.
Change-Id: I41e3ecfeae42d877ac5443a6bd622ac3d6c8120c
Reviewed-on: https://go-review.googlesource.com/c/go/+/185738
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
This reverts CL 151157.
CL 151157 introduced a crash when decoding into ",string" fields. It
came with a moderate speedup, so at this stage of the release cycle
let's just revert it, and reapply it in Go 1.14 with the fix in CL 190659.
Also applied the test cases from CL 190659.
Updates #33728
Change-Id: Ie46e2bc15224b251888580daf6b79d5865f3878e
Reviewed-on: https://go-review.googlesource.com/c/go/+/190909
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Currently test build fails with:
$ go test -tags=gofuzz encoding/json
encoding/json/fuzz.go:36:4: Println call has possible formatting directive %s
FAIL encoding/json [build failed]
Change-Id: I23aef44a421ed0e7bcf48b74ac5a8c6768a4841b
Reviewed-on: https://go-review.googlesource.com/c/go/+/190698
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
It wasn't obeyed in the case where the MarshalJSON method uses a pointer
receiver, and the encoder grabs the address of a value to find that
method. addrMarshalerEncoder is the function that does this work, but it
ignored opts.escapeHTML.
Here's the before and after of the added test case, which was failing
before the fix. Now the two cases are correct and consistent.
{"NonPtr":"<str>","Ptr":"\u003cstr\u003e"}
{"NonPtr":"<str>","Ptr":"<str>"}
Fixes#32896.
Change-Id: Idc53077ece074973558bd3bb5ad036380db0d02c
Reviewed-on: https://go-review.googlesource.com/c/go/+/184757
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Caleb Spare <cespare@gmail.com>
Shorten some of the longest tests that run during all.bash.
Removes 7r 50u 21s from all.bash.
After this change, all.bash is under 5 minutes again on my laptop.
For #26473.
Change-Id: Ie0460aa935808d65460408feaed210fbaa1d5d79
Reviewed-on: https://go-review.googlesource.com/c/go/+/177559
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Add Unwrap methods to types which wrap an underlying error:
"encodinc/csv".ParseError
"encoding/json".MarshalerError
"net/http".transportReadFromServerError
"net".OpError
"net".DNSConfigError
"net/url".Error
"os/exec".Error
"signal/internal/pty".PtyError
"text/template".ExecError
Add os.ErrTemporary. A case could be made for putting this error
value in package net, since no exported error types in package os
include a Temporary method. However, syscall errors returned from
the os package do include this method.
Add Is methods to error types with a Timeout or Temporary method,
making errors.Is(err, os.Err{Timeout,Temporary}) equivalent to
testing the corresponding method:
"context".DeadlineExceeded
"internal/poll".TimeoutError
"net".adrinfoErrno
"net".OpError
"net".DNSError
"net/http".httpError
"net/http".tlsHandshakeTimeoutError
"net/pipe".timeoutError
"net/url".Error
Updates #30322
Updates #29934
Change-Id: I409fb20c072ea39116ebfb8c7534d493483870dc
Reviewed-on: https://go-review.googlesource.com/c/go/+/170037
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
indirect walks down v until it gets to a non-pointer. But it does not
handle the case when v is a pointer to itself, like in:
var v interface{}
v = &v
Unmarshal(b, v)
So just stop immediately if we see v is a pointer to itself.
Fixes#31740
Change-Id: Ie396264119e24d70284cd9bf76dcb2050babb069
Reviewed-on: https://go-review.googlesource.com/c/go/+/174337
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Make explicit that Compact does HTML escaping.
Fixes#30357.
Change-Id: I4648f8f3e907d659db977d07253f716df6e07d7b
Reviewed-on: https://go-review.googlesource.com/c/go/+/173417
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
In the common case, structs have a handful of fields and most inputs
match struct field names exactly.
The previous code would do a linear search over the fields, stopping at
the first exact match, and otherwise using the first case insensitive
match.
This is unfortunate, because it means that for the common case, we'd do
a linear search with bytes.Equal. Even for structs with only two or
three fields, that is pretty wasteful.
Worse even, up until the exact match was found via the linear search,
all previous fields would run their equalFold functions, which aren't
cheap even in the simple case.
Instead, cache a map along with the field list that indexes the fields
by their name. This way, a case sensitive field search doesn't involve a
linear search, nor does it involve any equalFold func calls.
This patch should also slightly speed up cases where there's a case
insensitive match but not a case sensitive one, as then we'd avoid
calling bytes.Equal on all the fields. Though that's not a common case,
and there are no benchmarks for it.
name old time/op new time/op delta
CodeDecoder-8 11.0ms ± 0% 10.6ms ± 1% -4.42% (p=0.000 n=9+10)
name old speed new speed delta
CodeDecoder-8 176MB/s ± 0% 184MB/s ± 1% +4.62% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
CodeDecoder-8 2.28MB ± 0% 2.28MB ± 0% ~ (p=0.725 n=10+10)
name old allocs/op new allocs/op delta
CodeDecoder-8 76.9k ± 0% 76.9k ± 0% ~ (all equal)
Updates #28923.
Change-Id: I9929c1f06c76505e5b96914199315dbdaae5dc76
Reviewed-on: https://go-review.googlesource.com/c/go/+/172918
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
We can work out how many bytes can be unquoted trivially in
rescanLiteral, which already iterates over a string's bytes.
Removing the extra loop in unquoteBytes simplifies the function and
speeds it up, especially when decoding simple strings, which are common.
While at it, we can remove unnecessary checks like len(s)<2 and
s[0]=='"'. Add a comment explaining why.
name old time/op new time/op delta
CodeDecoder-8 11.2ms ± 0% 11.1ms ± 1% -1.63% (p=0.000 n=9+10)
name old speed new speed delta
CodeDecoder-8 173MB/s ± 0% 175MB/s ± 1% +1.66% (p=0.000 n=9+10)
Updates #28923.
Change-Id: I2436a3a7f8148a2f7a6a4cdbd7dec6b32ef5e20c
Reviewed-on: https://go-review.googlesource.com/c/go/+/151157
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
readValue is a hot function, clocking in at ~13% flat CPU use in
CodeDecoder. In particular, looping over the bytes is slow. That's
partially because the code contains a bounds check at the start of the
loop.
The source of the problem is that scanp is a signed integer, and comes
from a field, so the compiler doesn't know that it's non-negative. Help
it with a simple and comparatively cheap hint.
While at it, use scanp as the index variable directly, removing the need
for a duplicate index variable which is later added back into scanp.
name old time/op new time/op delta
CodeDecoder-8 11.3ms ± 1% 11.2ms ± 1% -0.98% (p=0.000 n=9+9)
name old speed new speed delta
CodeDecoder-8 172MB/s ± 1% 174MB/s ± 1% +0.99% (p=0.000 n=9+9)
Updates #28923.
Change-Id: I138f83babdf316fc97697cc18f595c3403c1ddb7
Reviewed-on: https://go-review.googlesource.com/c/go/+/170939
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This was the only benchmark missing the SetBytes call, as spotted
earlier by Bryan.
It's not required to make the benchmark useful, but it can still be a
good way to see how its speed is affected by the reduced allocations:
name time/op
CodeUnmarshal-8 12.1ms ± 1%
CodeUnmarshalReuse-8 11.4ms ± 1%
name speed
CodeUnmarshal-8 161MB/s ± 1%
CodeUnmarshalReuse-8 171MB/s ± 1%
name alloc/op
CodeUnmarshal-8 3.28MB ± 0%
CodeUnmarshalReuse-8 1.94MB ± 0%
name allocs/op
CodeUnmarshal-8 92.7k ± 0%
CodeUnmarshalReuse-8 77.6k ± 0%
While at it, remove some unnecessary empty lines.
Change-Id: Ib2bd92d5b3237b8f3092e8c6f863dab548fee2f5
Reviewed-on: https://go-review.googlesource.com/c/go/+/170938
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Decoder.Decode and Unmarshal actually scan the input bytes twice - the
first time to check for syntax errors and the length of the value, and
the second to perform the decoding.
It's in the second scan that we actually tokenize the bytes. Since
syntax errors aren't a possibility, we can take shortcuts.
In particular, literals such as quoted strings are very common in JSON,
so we can avoid a lot of work by special casing them.
name old time/op new time/op delta
CodeDecoder-8 10.3ms ± 1% 9.1ms ± 0% -11.89% (p=0.002 n=6+6)
UnicodeDecoder-8 342ns ± 0% 283ns ± 0% -17.25% (p=0.000 n=6+5)
DecoderStream-8 239ns ± 0% 230ns ± 0% -3.90% (p=0.000 n=6+5)
CodeUnmarshal-8 11.0ms ± 0% 9.8ms ± 0% -11.45% (p=0.002 n=6+6)
CodeUnmarshalReuse-8 10.3ms ± 0% 9.0ms ± 0% -12.72% (p=0.004 n=5+6)
UnmarshalString-8 104ns ± 0% 92ns ± 0% -11.35% (p=0.002 n=6+6)
UnmarshalFloat64-8 93.2ns ± 0% 87.6ns ± 0% -6.01% (p=0.010 n=6+4)
UnmarshalInt64-8 74.5ns ± 0% 71.5ns ± 0% -3.91% (p=0.000 n=5+6)
name old speed new speed delta
CodeDecoder-8 189MB/s ± 1% 214MB/s ± 0% +13.50% (p=0.002 n=6+6)
UnicodeDecoder-8 40.9MB/s ± 0% 49.5MB/s ± 0% +20.96% (p=0.002 n=6+6)
CodeUnmarshal-8 176MB/s ± 0% 199MB/s ± 0% +12.93% (p=0.002 n=6+6)
Updates #28923.
Change-Id: I7a5e2aef51bd4ddf2004aad24210f6f50e01eaeb
Reviewed-on: https://go-review.googlesource.com/c/go/+/151042
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
In golang.org/cl/145218, a feature was added where the JSON decoder
would keep track of the entire path to a field when reporting an
UnmarshalTypeError.
However, we all failed to check if this affected the benchmarks - myself
included, as a reviewer. Below are the numbers comparing the CL's parent
with itself, once it was merged:
name old time/op new time/op delta
CodeDecoder-8 12.9ms ± 1% 28.2ms ± 2% +119.33% (p=0.002 n=6+6)
name old speed new speed delta
CodeDecoder-8 151MB/s ± 1% 69MB/s ± 3% -54.40% (p=0.002 n=6+6)
name old alloc/op new alloc/op delta
CodeDecoder-8 2.74MB ± 0% 109.39MB ± 0% +3891.83% (p=0.002 n=6+6)
name old allocs/op new allocs/op delta
CodeDecoder-8 77.5k ± 0% 168.5k ± 0% +117.30% (p=0.004 n=6+5)
The reason why the decoder got twice as slow is because it now allocated
~40x as many objects, which puts a lot of pressure on the garbage
collector.
The reason is that the CL concatenated strings every time a nested field
was decoded. In other words, practically every field generated garbage
when decoded. This is hugely wasteful, especially considering that the
vast majority of JSON decoding inputs won't return UnmarshalTypeError.
Instead, use a stack of fields, and make sure to always use the same
backing array, to ensure we only need to grow the slice to the maximum
depth once.
The original CL also introduced a bug. The field stack string wasn't
reset to its original state when reaching "d.opcode == scanEndObject",
so the last field in a decoded struct could leak. For example, an added
test decodes a list of structs, and encoding/json before this CL would
fail:
got: cannot unmarshal string into Go struct field T.Ts.Y.Y.Y of type int
want: cannot unmarshal string into Go struct field T.Ts.Y of type int
To fix that, simply reset the stack after decoding every field, even if
it's the last.
Below is the original performance versus this CL. There's a tiny
performance hit, probably due to the append for every decoded field, but
at least we're back to the usual ~150MB/s.
name old time/op new time/op delta
CodeDecoder-8 12.9ms ± 1% 13.0ms ± 1% +1.25% (p=0.009 n=6+6)
name old speed new speed delta
CodeDecoder-8 151MB/s ± 1% 149MB/s ± 1% -1.24% (p=0.009 n=6+6)
name old alloc/op new alloc/op delta
CodeDecoder-8 2.74MB ± 0% 2.74MB ± 0% +0.00% (p=0.002 n=6+6)
name old allocs/op new allocs/op delta
CodeDecoder-8 77.5k ± 0% 77.5k ± 0% +0.00% (p=0.002 n=6+6)
Finally, make all of these benchmarks report allocs by default. The
decoder ones are pretty sensitive to generated garbage, so ReportAllocs
would have made the performance regression more obvious.
Change-Id: I67b50f86b2e72f55539429450c67bfb1a9464b67
Reviewed-on: https://go-review.googlesource.com/c/go/+/167978
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
If a for loop has a simple condition and begins with a simple
"if x { break; }"; we can simply add "!x" to the loop's condition.
While at it, simplify a few assignments to use the common pattern
"x := staticDefault; if cond { x = otherValue(); }".
Finally, simplify a couple of var declarations.
Change-Id: I413982c6abd32905adc85a9a666cb3819139c19f
Reviewed-on: https://go-review.googlesource.com/c/go/+/165342
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
I had been finding these over a year or so, but none were big enough
changes to warrant CLs. They're a handful now, so clean them all up in a
single commit.
The smaller bodies get a bit simpler, but most importantly, the larger
bodies get unindented.
Change-Id: I5707a6fee27d4c9ff9efd3d363af575d7a4bf2aa
Reviewed-on: https://go-review.googlesource.com/c/go/+/165340
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
When parsing nested object, UnmarshalTypeError does not contain actual
path to nested field in original JSON.
This commit change Field to contain the full path to that field. One
can get the Field name by stripping all the leading path elements.
Fixes#22369
Change-Id: I6969cc08abe8387a351e3fb2944adfaa0dccad2a
Reviewed-on: https://go-review.googlesource.com/c/go/+/145218
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Comparing errors using DeepEqual breaks if frame information
is added as proposed in Issue #29934.
Updates #29934.
Change-Id: Ib430c9ddbe588dd1dd51314c408c74c07285e1ff
Reviewed-on: https://go-review.googlesource.com/c/162179
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Both the encoding/xml and encoding/json packages support custom
marshalers for JSON and XML, as well as the basic encoding.TextMarshaler
and encoding.TextUnmarshaler interfaces, but the docs and examples for
these are missing.
There are docs for how to use encoding.TextMarshaler and
encoding.TextUnmarshaler in encoding/json, but not encoding/xml. There
are no examples for how to use them with either json or xml. This commit
includes docs for encoding/xml and examples for both encoding/json and
encoding/xml.
There is an example using custom marshalers MarshalJSON and
UnmarshalJSON in encoding/json, but not MarshalXML and UnmarshalXML in
encoding/json. These docs are more so necessary for encoding/xml because
the complexities of XML documents is significantly greater than JSON
documents which more often leads to the need for custom marshaling. The
encoding/json package includes an example of how to write a custom
marshaler, and this commit includes the same example for the xml
package.
All examples are mirrored off the existing custom marshaler example in
encoding/json.
Fixes#6859
Change-Id: Ic93abc27c0b4d5e48dea6ede4e20b1bedca4ab39
Reviewed-on: https://go-review.googlesource.com/c/76350
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Calling .Interface on a struct field's reflect.Value isn't always safe.
For example, if that field is an unexported anonymous struct.
We only descended into this branch if the struct type had any methods,
so this bug had gone unnoticed for a few release cycles.
Add the check, and add a simple test case.
Fixes#28145.
Change-Id: I02f7e0ab9a4a0c18a5e2164211922fe9c3d30f64
Reviewed-on: https://go-review.googlesource.com/c/141537
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Given a program as follows:
data := []byte(`{"F": {
"a": 2,
"3": 4
}}`)
json.Unmarshal(data, &map[string]map[int]int{})
The JSON package should error, as "a" is not a valid integer. However,
we'd encounter a panic:
panic: JSON decoder out of sync - data changing underfoot?
The reason was that decodeState.object would return a nil error on
encountering the invalid map key string, while saving the key type error
for later. This broke if we were inside another object, as we would
abruptly end parsing the nested object, leaving the decoder in an
unexpected state.
To fix this, simply avoid storing the map element and continue decoding
the object, to leave the decoder state exactly as if we hadn't seen an
invalid key type.
This affected both signed and unsigned integer keys, so fix both and add
two test cases.
Updates #28189.
Change-Id: I8a6204cc3ff9fb04ed769df7a20a824c8b94faff
Reviewed-on: https://go-review.googlesource.com/c/142518
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This change makes stateEndTop use isSpace instead of specifically
recreating the same functionality.
Change-Id: I81f8f51682e46e7f8e2b9fed423a968457200625
Reviewed-on: https://go-review.googlesource.com/c/121797
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Having these panic-like errors used to be ok, since they were used in
the internal decoder state instead of passed around via return
parameters.
Recently, the decoder was rewritten to use explicit error returns
instead. This error is a terrible fit for error returns; a handful of
functions must return an error because of it, and their callers must
check for an error that should never happen.
This is precisely what panics are for, so use them. The test coverage of
the package goes up from 91.3% to 91.6%, and performance is unaffected.
We can also get rid of some unnecessary verbosity in the code.
name old time/op new time/op delta
CodeDecoder-4 27.5ms ± 1% 27.5ms ± 1% ~ (p=0.937 n=6+6)
Change-Id: I01033b3f5b7c0cf0985082fa272754f96bf6353c
Reviewed-on: https://go-review.googlesource.com/134835
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
The overall coverage of the json package goes up from 90.8% to 91.3%.
While at it, apply two minor code simplifications found while inspecting
the HTML coverage report.
Change-Id: I0fba968afeedc813b1385e4bde72d93b878854d7
Reviewed-on: https://go-review.googlesource.com/134735
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Given the following types:
type S2 struct{ Field string }
type S struct{ *S2 }
Marshalling a value of type T1 should result in "{}", as there's no way
to access any value of T2.Field. This is how Go 1.10 and earlier
versions behave.
However, in the recent refactor golang.org/cl/125417 I broke this logic.
When the encoder found an anonymous struct pointer field that was nil,
it no longer skipped the embedded fields underneath it. This can be seen
in the added test:
--- FAIL: TestAnonymousFields/EmbeddedFieldBehindNilPointer (0.00s)
encode_test.go:430: Marshal() = "{\"Field\":\"\\u003c*json.S2 Value\\u003e\"}", want "{}"
The human error was a misplaced label, meaning we weren't actually
skipping the right loop iteration. Fix that.
Change-Id: Iba8a4a77d358dac73dcba4018498fe4f81afa263
Reviewed-on: https://go-review.googlesource.com/131376
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TestRawMessage now passes without the need for the RawMessage field to
be a pointer. The TODO dates all the way back to 2010, so I presume the
issue has since been fixed.
TestNullRawMessage tested the decoding of a JSON null into a
*RawMessage. The existing behavior was correct, but for the sake of
completeness a non-pointer RawMessage field has been added too. The
non-pointer field behaves differently, as one can read in the docs:
To unmarshal JSON into a value implementing the Unmarshaler
interface, Unmarshal calls that value's UnmarshalJSON method,
including when the input is a JSON null.
Change-Id: Iabaed75d4ed10ea427d135ee1b80c6e6b83b2e6e
Reviewed-on: https://go-review.googlesource.com/131377
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Encoders like map and array can use the much cheaper "i > 0" check to
see if we're not writing the first element. However, since struct fields
support omitempty, we need to keep track of that separately.
This is much more expensive - after calling the field encoder itself,
and retrieving the field via reflection, this branch was the third most
expensive piece of this field loop.
Instead, hoist the branch logic outside of the loop. The code doesn't
get much more complex, since we just delay the writing of each byte
until the next iteration. Yet the performance improvement is noticeable,
even when the struct types in CodeEncoder only have 2 and 7 fields,
respectively.
name old time/op new time/op delta
CodeEncoder-4 5.39ms ± 0% 5.31ms ± 0% -1.37% (p=0.010 n=4+6)
name old speed new speed delta
CodeEncoder-4 360MB/s ± 0% 365MB/s ± 0% +1.39% (p=0.010 n=4+6)
Updates #5683.
Change-Id: I2662cf459e0dfd68e56fa52bc898a417e84266c2
Reviewed-on: https://go-review.googlesource.com/131401
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
A few encoder struct types, such as map and slice, only encapsulate
other prepared encoder funcs. Using pointer receivers has no advantage,
and makes calling these methods slightly more expensive.
Not a huge performance win, but certainly an easy one. The struct types
used in the benchmark below contain one slice field and one pointer
field.
name old time/op new time/op delta
CodeEncoder-4 5.48ms ± 0% 5.39ms ± 0% -1.66% (p=0.010 n=6+4)
name old speed new speed delta
CodeEncoder-4 354MB/s ± 0% 360MB/s ± 0% +1.69% (p=0.010 n=6+4)
Updates #5683.
Change-Id: I9f78dbe07fcc6fbf19a6d96c22f5d6970db9eca4
Reviewed-on: https://go-review.googlesource.com/131400
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Some WriteByte('\\') calls can be deduplicated.
fillField is used in two occasions, but it is unnecessary when adding
fields to the "next" stack, as those aren't used for the final encoding.
Inline the func with its only remaining call.
Finally, unindent a default-if block.
The performance of the encoder is unaffected:
name old time/op new time/op delta
CodeEncoder-4 6.65ms ± 1% 6.65ms ± 0% ~ (p=0.662 n=6+5)
Change-Id: Ie55baeab89abad9b9f13e9f6ca886a670c30dba9
Reviewed-on: https://go-review.googlesource.com/122461
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This function was only used in a single place - in the field encoding
loop within the struct encoder.
Inlining the function call manually lets us get rid of the call
overhead. But most importantly, it lets us simplify the logic afterward.
We no longer need to use reflect.Value{} and !fv.IsValid(), as we can
skip the field immediately.
The two factors combined (mostly just the latter) give a moderate speed
improvement to this hot loop.
name old time/op new time/op delta
CodeEncoder-4 6.01ms ± 1% 5.91ms ± 1% -1.66% (p=0.002 n=6+6)
name old speed new speed delta
CodeEncoder-4 323MB/s ± 1% 328MB/s ± 1% +1.69% (p=0.002 n=6+6)
Updates #5683.
Change-Id: I12757c325a68abb2856026cf719c122612a1f38e
Reviewed-on: https://go-review.googlesource.com/125417
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
structEncoder had two slices - the list of fields, and a list containing
the encoder for each field. structEncoder.encode then looped over the
fields, and indexed into the second slice to grab the field encoder.
However, this makes it very hard for the compiler to be able to prove
that the two slices always have the same length, and that the index
expression doesn't need a bounds check.
Merge the two slices into one to completely remove the need for bounds
checks in the hot loop.
While at it, don't copy the field elements when ranging, which greatly
speeds up the hot loop in structEncoder.
name old time/op new time/op delta
CodeEncoder-4 6.18ms ± 0% 5.56ms ± 0% -10.08% (p=0.002 n=6+6)
name old speed new speed delta
CodeEncoder-4 314MB/s ± 0% 349MB/s ± 0% +11.21% (p=0.002 n=6+6)
name old alloc/op new alloc/op delta
CodeEncoder-4 93.2kB ± 0% 62.1kB ± 0% -33.33% (p=0.002 n=6+6)
Updates #5683.
Change-Id: I0dd47783530f439b125e084aede09dda172eb1e8
Reviewed-on: https://go-review.googlesource.com/125416
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reuse v.Type() and cachedTypeFields(t) when decoding maps and structs.
Always use the same data slices when in hot loops, to ensure that the
compiler generates good code. "for i < len(data) { use(d.data[i]) }"
makes it harder for the compiler.
Finally, do other minor clean-ups, such as deduplicating switch cases,
and using a switch instead of three chained ifs.
The decoder sees a noticeable speed-up, in particular when decoding
structs.
name old time/op new time/op delta
CodeDecoder-4 29.8ms ± 1% 27.5ms ± 0% -7.83% (p=0.002 n=6+6)
name old speed new speed delta
CodeDecoder-4 65.0MB/s ± 1% 70.6MB/s ± 0% +8.49% (p=0.002 n=6+6)
Updates #5683.
Change-Id: I9d751e22502221962da696e48996ffdeb777277d
Reviewed-on: https://go-review.googlesource.com/122468
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Calling Name on a reflect.Type is somewhat expensive, as it involves a
number of nested calls and string handling.
This cost was showing up when decoding structs, as we were calling it to
set up an error context.
We can avoid the extra work unless we do encounter an error, which makes
decoding via struct types faster.
name old time/op new time/op delta
CodeDecoder-4 31.0ms ± 1% 29.9ms ± 1% -3.69% (p=0.002 n=6+6)
name old speed new speed delta
CodeDecoder-4 62.6MB/s ± 1% 65.0MB/s ± 1% +3.83% (p=0.002 n=6+6)
Updates #5683.
Change-Id: I48a3a85ef0ba96f524b7c3e9096cb2c4589e077a
Reviewed-on: https://go-review.googlesource.com/122467
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
If the encoded bytes fit in the bootstrap array encodeState.scratch, use
that instead of allocating a new byte slice.
Also tweaked the Encoding vs Encoder heuristic to use the length of the
encoded bytes, not the length of the input bytes. Encoding is used for
allocations of up to 1024 bytes, as we measured 2048 to be the point
where it no longer provides a noticeable advantage.
Also added some benchmarks. Only the first case changes in behavior.
name old time/op new time/op delta
MarshalBytes/32-4 420ns ± 1% 383ns ± 1% -8.69% (p=0.002 n=6+6)
MarshalBytes/256-4 913ns ± 1% 915ns ± 0% ~ (p=0.580 n=5+6)
MarshalBytes/4096-4 7.72µs ± 0% 7.74µs ± 0% ~ (p=0.340 n=5+6)
name old alloc/op new alloc/op delta
MarshalBytes/32-4 112B ± 0% 64B ± 0% -42.86% (p=0.002 n=6+6)
MarshalBytes/256-4 736B ± 0% 736B ± 0% ~ (all equal)
MarshalBytes/4096-4 7.30kB ± 0% 7.30kB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
MarshalBytes/32-4 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.002 n=6+6)
MarshalBytes/256-4 2.00 ± 0% 2.00 ± 0% ~ (all equal)
MarshalBytes/4096-4 2.00 ± 0% 2.00 ± 0% ~ (all equal)
Updates #5683.
Change-Id: I5fa55c27bd7728338d770ae7c0756885ba9a5724
Reviewed-on: https://go-review.googlesource.com/122462
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Struct field names are static, so we can run HTMLEscape on them when
building each struct type encoder. Then, when running the struct
encoder, we can select either the original or the escaped field name to
write directly.
When the encoder is not escaping HTML, using the original string works
because neither Go struct field names nor JSON tags allow any characters
that would need to be escaped, like '"', '\\', or '\n'.
When the encoder is escaping HTML, the only difference is that '<', '>',
and '&' are allowed via JSON struct field tags, hence why we use
HTMLEscape to properly escape them.
All of the above lets us encode field names with a simple if/else and
WriteString calls, which are considerably simpler and faster than
encoding an arbitrary string.
While at it, also include the quotes and colon in these strings, to
avoid three WriteByte calls in the loop hot path.
Also added a few tests, to ensure that the behavior in these edge cases
is not broken. The output of the tests is the same if this optimization
is reverted.
name old time/op new time/op delta
CodeEncoder-4 7.12ms ± 0% 6.14ms ± 0% -13.85% (p=0.004 n=6+5)
name old speed new speed delta
CodeEncoder-4 272MB/s ± 0% 316MB/s ± 0% +16.08% (p=0.004 n=6+5)
name old alloc/op new alloc/op delta
CodeEncoder-4 91.9kB ± 0% 93.2kB ± 0% +1.43% (p=0.002 n=6+6)
name old allocs/op new allocs/op delta
CodeEncoder-4 0.00 0.00 ~ (all equal)
Updates #5683.
Change-Id: I6f6a340d0de4670799ce38cf95b2092822d2e3ef
Reviewed-on: https://go-review.googlesource.com/122460
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The existing Decoder.tokenError implementation creates its error messages by
concatenating "invalid character " + quoteChar(c) + " " + context. All context
values however already start with a space leading to error messages containing
two spaces.
This change removes " " from the concatenation expression.
Fixes#26587
Change-Id: I93d14319396636b2a40d55053bda88c98e94a81a
GitHub-Last-Rev: 6db7e1991b
GitHub-Pull-Request: golang/go#26588
Reviewed-on: https://go-review.googlesource.com/125775
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Each URL was manually verified to ensure it did not serve up incorrect
content.
Change-Id: I4dc846227af95a73ee9a3074d0c379ff0fa955df
Reviewed-on: https://go-review.googlesource.com/115798
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Fixes#18037
Change-Id: I20e27bcc013b00b726eb348daf5ca86b138ddcc2
Reviewed-on: https://go-review.googlesource.com/107598
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Fixes golint warning about "if block ends with a return statement, so drop this else and outdent its block".
Change-Id: Id17ad0bf37ba939386b177b709e9e3c067d8ba21
Reviewed-on: https://go-review.googlesource.com/105736
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The go/printer (and thus gofmt) uses a heuristic to determine
whether to break alignment between elements of an expression
list which is spread across multiple lines. The heuristic only
kicked in if the entry sizes (character length) was above a
certain threshold (20) and the ratio between the previous and
current entry size was above a certain value (4).
This heuristic worked reasonably most of the time, but also
led to unfortunate breaks in many cases where a single entry
was suddenly much smaller (or larger) then the previous one.
The behavior of gofmt was sufficiently mysterious in some of
these situations that many issues were filed against it.
The simplest solution to address this problem is to remove
the heuristic altogether and have a programmer introduce
empty lines to force different alignments if it improves
readability. The problem with that approach is that the
places where it really matters, very long tables with many
(hundreds, or more) entries, may be machine-generated and
not "post-processed" by a human (e.g., unicode/utf8/tables.go).
If a single one of those entries is overlong, the result
would be that the alignment would force all comments or
values in key:value pairs to be adjusted to that overlong
value, making the table hard to read (e.g., that entry may
not even be visible on screen and all other entries seem
spaced out too wide).
Instead, we opted for a slightly improved heuristic that
behaves much better for "normal", human-written code.
1) The threshold is increased from 20 to 40. This disables
the heuristic for many common cases yet even if the alignment
is not "ideal", 40 is not that many characters per line with
todays screens, making it very likely that the entire line
remains "visible" in an editor.
2) Changed the heuristic to not simply look at the size ratio
between current and previous line, but instead considering the
geometric mean of the sizes of the previous (aligned) lines.
This emphasizes the "overall picture" of the previous lines,
rather than a single one (which might be an outlier).
3) Changed the ratio from 4 to 2.5. Now that we ignore sizes
below 40, a ratio of 4 would mean that a new entry would have
to be 4 times bigger (160) or smaller (10) before alignment
would be broken. A ratio of 2.5 seems more sensible.
Applied updated gofmt to all of src and misc. Also tested
against several former issues that complained about this
and verified that the output for the given examples is
satisfactory (added respective test cases).
Some of the files changed because they were not gofmt-ed
in the first place.
For #644.
For #7335.
For #10392.
(and probably more related issues)
Fixes#22852.
Change-Id: I5e48b3d3b157a5cf2d649833b7297b33f43a6f6e
The previous type cache is quadratic in time in the situation where
new types are continually encountered. Now that it is possible to dynamically
create new types with the reflect package, this can cause json to
perform very poorly.
Switch to sync.Map which does well when the cache has hit steady state,
but also handles occasional updates in better than quadratic time.
benchmark old ns/op new ns/op delta
BenchmarkTypeFieldsCache/MissTypes1-8 14817 16202 +9.35%
BenchmarkTypeFieldsCache/MissTypes10-8 70926 69144 -2.51%
BenchmarkTypeFieldsCache/MissTypes100-8 976467 208973 -78.60%
BenchmarkTypeFieldsCache/MissTypes1000-8 79520162 1750371 -97.80%
BenchmarkTypeFieldsCache/MissTypes10000-8 6873625837 16847806 -99.75%
BenchmarkTypeFieldsCache/HitTypes1000-8 7.51 8.80 +17.18%
BenchmarkTypeFieldsCache/HitTypes10000-8 7.58 8.68 +14.51%
The old implementation takes 12 minutes just to build a cache of size 1e5
due to the quadratic behavior. I did not bother benchmark sizes above that.
Change-Id: I5e6facc1eb8e1b80e5ca285e4dd2cc8815618dad
Reviewed-on: https://go-review.googlesource.com/76850
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Consider the following:
type child struct{ Field string }
type parent struct{ child }
p := new(parent)
v := reflect.ValueOf(p).Elem().Field(0)
v.Field(0).SetString("hello") // v.Field = "hello"
v = v.Addr().Elem() // v = *(&v)
v.Field(0).SetString("goodbye") // v.Field = "goodbye"
It would appear that v.Addr().Elem() should have the same value, and
that it would be safe to set "goodbye".
However, after CL 66331, any interspersed calls between Field calls
causes the RO flag to be set.
Thus, setting to "goodbye" actually causes a panic.
That CL affects decodeState.indirect which assumes that back-to-back
Value.Addr().Elem() is side-effect free. We fix that logic to keep
track of the Addr() and Elem() calls and set v back to the original
after a full round-trip has occured.
Fixes#24152
Updates #24153
Change-Id: Ie50f8fe963f00cef8515d89d1d5cbc43b76d9f9c
Reviewed-on: https://go-review.googlesource.com/97796
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Rather than only ignoring runtime.Error panics, which are a very
narrow set of possible panic values, switch it such that the json
package only captures panic values that have been properly wrapped
in a jsonError struct. This ensures that only intentional panics
originating from the json package are captured.
Fixes#23012
Change-Id: I5e85200259edd2abb1b0512ce6cc288849151a6d
Reviewed-on: https://go-review.googlesource.com/94019
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This CL reverts CL 76851 and takes a different approach to #21357.
The changes in encode.go and encode_test.go are reverts that
rolls back the changed behavior in CL 76851 where
embedded pointers to unexported struct types were
unilaterally ignored in both marshal and unmarshal.
Instead, these fields are handled as before with the exception that
it returns an error when Unmarshal is unable to set an unexported field.
The behavior of Marshal is now unchanged with regards to #21357.
This policy maintains the greatest degree of backwards compatibility
and avoids silently discarding data the user may have expected to be present.
Fixes#21357
Change-Id: I7dc753280c99f786ac51acf7e6c0246618c8b2b1
Reviewed-on: https://go-review.googlesource.com/82135
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
It was added in CL 79995. It is unnecessarily confusing.
Change-Id: Ib8ff35b9f71b54ff99d2d6e0534c7128e1f4345a
Reviewed-on: https://go-review.googlesource.com/80035
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
CL 60410 fixes a bug in reflect that allows assignments to an embedded
field of a pointer to an unexported struct type.
This breaks the json package because unmarshal is now unable to assign
a newly allocated struct to such fields.
In order to be consistent in the behavior for marshal and unmarshal,
this CL changes both marshal and unmarshal to always ignore
embedded pointers to unexported structs.
Fixes#21357
Change-Id: If62ea11155555e61115ebb9cfa5305caf101bde5
Reviewed-on: https://go-review.googlesource.com/76851
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Kevin Burke
Russ Cox
Carlos Alexandro Becker
lujjjh
Sean Liao
Damien Neil
Daniel Martí
codesoap
Jordan Liggitt
Jon Johnson
Phil Pearl
Eugene Kalinin
Francesco Renzi
Cuong Manh Le
Tom Thorogood
Lucas Bremgartner
William Poussier
andig
Romain Baugue
Ainar Garipov
Eli Bendersky
itchyny
Filippo Valsorda
Dmitry Vyukov
Marcel van Lohuizen
Ketan Parmar
GuilhermeCaruso
Leigh McCulloch
hearot
Gabriel Aszalos
Ian Davis
Taesu Pyo
Tim Cooper
Philip Børgesen
tengufromsky
Michael Fraenkel
Jeremy Jackins
bontequero
Robert Griesemer
Pascal S. de Kloe
Joe Tsai
rajender
Michael Schurter