Commit Graph

394 Commits

Author SHA1 Message Date
Brad Fitzpatrick 3813edf26e all: use "reports whether" consistently in the few places that didn't
Go documentation style for boolean funcs is to say:

    // Foo reports whether ...
    func Foo() bool

(rather than "returns true if")

This CL also replaces 4 uses of "iff" with the same "reports whether"
wording, which doesn't lose any meaning, and will prevent people from
sending typo fixes when they don't realize it's "if and only if". In
the past I think we've had the typo CLs updated to just say "reports
whether". So do them all at once.

(Inspired by the addition of another "returns true if" in CL 146938
in fd_plan9.go)

Created with:

$ perl -i -npe 's/returns true if/reports whether/' $(git grep -l "returns true iff" | grep -v vendor)
$ perl -i -npe 's/returns true if/reports whether/' $(git grep -l "returns true if" | grep -v vendor)

Change-Id: Ided502237f5ab0d25cb625dbab12529c361a8b9f
Reviewed-on: https://go-review.googlesource.com/c/147037
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-11-02 22:47:58 +00:00
hearot 3fe9d4c1c8 encoding/json: add example to Valid
Change-Id: I411483d76a2ca91cd15ff42ae1adb9134486d183
Reviewed-on: https://go-review.googlesource.com/c/145278
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-10-28 19:49:25 +00:00
Rob Pike 956af97880 encoding/gob: delete out of memory test
Now that the library allows much larger data, it can kill
machines with less memory.

Fixes #28321

Change-Id: I98e1a5fdf812fd75adfb22bf01542423de405fe2
Reviewed-on: https://go-review.googlesource.com/c/143817
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2018-10-22 22:23:03 +00:00
Rob Pike 3bf9b77c0c encoding/gob: increase "tooBig" from 1GB to 8GB on 64-bit machines
A little shift magic makes it easy to adjust the maximum buffer
size on machines with larger integers.

Fixes #27635

Change-Id: I1f26b07a363fbb9730df2377052475fa88bbb781
Reviewed-on: https://go-review.googlesource.com/c/143678
Run-TryBot: Rob Pike <r@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-10-22 05:45:18 +00:00
Daniel Martí 4b36e129f8 encoding/json: always verify we can get a field's value
Calling .Interface on a struct field's reflect.Value isn't always safe.
For example, if that field is an unexported anonymous struct.

We only descended into this branch if the struct type had any methods,
so this bug had gone unnoticed for a few release cycles.

Add the check, and add a simple test case.

Fixes #28145.

Change-Id: I02f7e0ab9a4a0c18a5e2164211922fe9c3d30f64
Reviewed-on: https://go-review.googlesource.com/c/141537
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-10-16 14:02:52 +00:00
Daniel Martí 5eff6bfdbc encoding/json: fix "data changed underfoot?" panic
Given a program as follows:

	data := []byte(`{"F": {
		"a": 2,
		"3": 4
	}}`)
	json.Unmarshal(data, &map[string]map[int]int{})

The JSON package should error, as "a" is not a valid integer. However,
we'd encounter a panic:

	panic: JSON decoder out of sync - data changing underfoot?

The reason was that decodeState.object would return a nil error on
encountering the invalid map key string, while saving the key type error
for later. This broke if we were inside another object, as we would
abruptly end parsing the nested object, leaving the decoder in an
unexpected state.

To fix this, simply avoid storing the map element and continue decoding
the object, to leave the decoder state exactly as if we hadn't seen an
invalid key type.

This affected both signed and unsigned integer keys, so fix both and add
two test cases.

Updates #28189.

Change-Id: I8a6204cc3ff9fb04ed769df7a20a824c8b94faff
Reviewed-on: https://go-review.googlesource.com/c/142518
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-10-16 13:45:08 +00:00
Mihai Todor 1f95e0a9d9 encoding/base64: fix typo in decodeQuantum docs
Change-Id: I643540bcea574d8a70b79237d97097dcc7368766
GitHub-Last-Rev: e2be58d1ab
GitHub-Pull-Request: golang/go#28125
Reviewed-on: https://go-review.googlesource.com/c/141119
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-12 14:36:03 +00:00
Gabriel Aszalos 9f193fbe31 encoding/json: use isSpace in stateEndTop
This change makes stateEndTop use isSpace instead of specifically
recreating the same functionality.

Change-Id: I81f8f51682e46e7f8e2b9fed423a968457200625
Reviewed-on: https://go-review.googlesource.com/c/121797
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-10-06 15:40:02 +00:00
Brad Fitzpatrick da0d1a44ba all: use strings.ReplaceAll and bytes.ReplaceAll where applicable
I omitted vendor directories and anything necessary for bootstrapping.
(Tested by bootstrapping with Go 1.4)

Updates #27864

Change-Id: I7d9b68d0372d3a34dee22966cca323513ece7e8a
Reviewed-on: https://go-review.googlesource.com/137856
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-09-26 22:14:25 +00:00
Daniel Martí b2fcfc1a50 encoding/json: use panics for phase errors
Having these panic-like errors used to be ok, since they were used in
the internal decoder state instead of passed around via return
parameters.

Recently, the decoder was rewritten to use explicit error returns
instead. This error is a terrible fit for error returns; a handful of
functions must return an error because of it, and their callers must
check for an error that should never happen.

This is precisely what panics are for, so use them. The test coverage of
the package goes up from 91.3% to 91.6%, and performance is unaffected.
We can also get rid of some unnecessary verbosity in the code.

name           old time/op    new time/op    delta
CodeDecoder-4    27.5ms ± 1%    27.5ms ± 1%   ~     (p=0.937 n=6+6)

Change-Id: I01033b3f5b7c0cf0985082fa272754f96bf6353c
Reviewed-on: https://go-review.googlesource.com/134835
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2018-09-12 07:50:10 +00:00
Daniel Martí dc3680865a encoding/json: more tests to cover decoding edge cases
The overall coverage of the json package goes up from 90.8% to 91.3%.

While at it, apply two minor code simplifications found while inspecting
the HTML coverage report.

Change-Id: I0fba968afeedc813b1385e4bde72d93b878854d7
Reviewed-on: https://go-review.googlesource.com/134735
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-09-12 06:48:09 +00:00
Ian Davis 22afb3571c encoding/json: recover saved error context when unmarshalling
Fixes: #27464

Change-Id: I270c56fd0d5ae8787a1293029aff3072f4f52f33
Reviewed-on: https://go-review.googlesource.com/132955
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-09-06 08:38:44 +00:00
Josh Bleecher Snyder 2179e495ce encoding/binary: simplify Read and Write
There's no need to manually manage the backing slice for bs.
Removing it simplifies the code, removes some allocations,
and speeds it up slightly.

Fixes #27403

name                     old time/op    new time/op    delta
ReadSlice1000Int32s-8      6.39µs ± 1%    6.31µs ± 1%   -1.37%  (p=0.000 n=27+27)
ReadStruct-8               1.25µs ± 2%    1.23µs ± 2%   -1.06%  (p=0.003 n=30+29)
ReadInts-8                  301ns ± 0%     297ns ± 1%   -1.21%  (p=0.000 n=27+30)
WriteInts-8                 325ns ± 1%     320ns ± 1%   -1.59%  (p=0.000 n=26+29)
WriteSlice1000Int32s-8     6.60µs ± 0%    6.52µs ± 0%   -1.23%  (p=0.000 n=28+27)
PutUint16-8                0.72ns ± 2%    0.71ns ± 2%     ~     (p=0.286 n=30+30)
PutUint32-8                0.71ns ± 1%    0.71ns ± 0%   -0.42%  (p=0.003 n=30+25)
PutUint64-8                0.78ns ± 2%    0.78ns ± 0%   -0.55%  (p=0.001 n=30+27)
LittleEndianPutUint16-8    0.57ns ± 0%    0.57ns ± 0%     ~     (all equal)
LittleEndianPutUint32-8    0.57ns ± 0%    0.57ns ± 0%     ~     (all equal)
LittleEndianPutUint64-8    0.57ns ± 0%    0.57ns ± 0%     ~     (all equal)
PutUvarint32-8             23.1ns ± 1%    23.1ns ± 1%     ~     (p=0.925 n=26+29)
PutUvarint64-8             57.5ns ± 2%    57.3ns ± 1%     ~     (p=0.338 n=30+26)
[Geo mean]                 23.0ns         22.9ns        -0.61%

name                     old speed      new speed      delta
ReadSlice1000Int32s-8     626MB/s ± 1%   634MB/s ± 1%   +1.38%  (p=0.000 n=27+27)
ReadStruct-8             60.2MB/s ± 2%  60.8MB/s ± 2%   +1.08%  (p=0.002 n=30+29)
ReadInts-8                100MB/s ± 1%   101MB/s ± 1%   +1.24%  (p=0.000 n=27+30)
WriteInts-8              92.2MB/s ± 1%  93.6MB/s ± 1%   +1.56%  (p=0.000 n=26+29)
WriteSlice1000Int32s-8    606MB/s ± 0%   614MB/s ± 0%   +1.24%  (p=0.000 n=28+27)
PutUint16-8              2.80GB/s ± 1%  2.80GB/s ± 1%     ~     (p=0.095 n=28+29)
PutUint32-8              5.61GB/s ± 1%  5.62GB/s ± 1%     ~     (p=0.069 n=27+28)
PutUint64-8              10.2GB/s ± 1%  10.2GB/s ± 0%   +0.15%  (p=0.039 n=27+27)
LittleEndianPutUint16-8  3.50GB/s ± 1%  3.50GB/s ± 1%     ~     (p=0.552 n=30+29)
LittleEndianPutUint32-8  7.01GB/s ± 1%  7.02GB/s ± 1%     ~     (p=0.160 n=29+27)
LittleEndianPutUint64-8  14.0GB/s ± 1%  14.0GB/s ± 1%     ~     (p=0.413 n=29+29)
PutUvarint32-8            174MB/s ± 1%   173MB/s ± 1%     ~     (p=0.648 n=25+30)
PutUvarint64-8            139MB/s ± 2%   140MB/s ± 1%     ~     (p=0.271 n=30+26)
[Geo mean]                906MB/s        911MB/s        +0.55%

name                     old alloc/op   new alloc/op   delta
ReadSlice1000Int32s-8      4.14kB ± 0%    4.13kB ± 0%   -0.19%  (p=0.000 n=30+30)
ReadStruct-8                 200B ± 0%      200B ± 0%     ~     (all equal)
ReadInts-8                  64.0B ± 0%     32.0B ± 0%  -50.00%  (p=0.000 n=30+30)
WriteInts-8                  112B ± 0%       64B ± 0%  -42.86%  (p=0.000 n=30+30)
WriteSlice1000Int32s-8     4.14kB ± 0%    4.13kB ± 0%   -0.19%  (p=0.000 n=30+30)
PutUint16-8                 0.00B          0.00B          ~     (all equal)
PutUint32-8                 0.00B          0.00B          ~     (all equal)
PutUint64-8                 0.00B          0.00B          ~     (all equal)
LittleEndianPutUint16-8     0.00B          0.00B          ~     (all equal)
LittleEndianPutUint32-8     0.00B          0.00B          ~     (all equal)
LittleEndianPutUint64-8     0.00B          0.00B          ~     (all equal)
PutUvarint32-8              0.00B          0.00B          ~     (all equal)
PutUvarint64-8              0.00B          0.00B          ~     (all equal)
[Geo mean]                   476B           370B       -22.22%

name                     old allocs/op  new allocs/op  delta
ReadSlice1000Int32s-8        3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=30+30)
ReadStruct-8                 16.0 ± 0%      16.0 ± 0%     ~     (all equal)
ReadInts-8                   8.00 ± 0%      8.00 ± 0%     ~     (all equal)
WriteInts-8                  14.0 ± 0%      14.0 ± 0%     ~     (all equal)
WriteSlice1000Int32s-8       3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=30+30)
PutUint16-8                  0.00           0.00          ~     (all equal)
PutUint32-8                  0.00           0.00          ~     (all equal)
PutUint64-8                  0.00           0.00          ~     (all equal)
LittleEndianPutUint16-8      0.00           0.00          ~     (all equal)
LittleEndianPutUint32-8      0.00           0.00          ~     (all equal)
LittleEndianPutUint64-8      0.00           0.00          ~     (all equal)
PutUvarint32-8               0.00           0.00          ~     (all equal)
PutUvarint64-8               0.00           0.00          ~     (all equal)
[Geo mean]                   6.94           5.90       -14.97%

Change-Id: I3790b93e4190d98621d5f2c47e42929a18f56c2e
Reviewed-on: https://go-review.googlesource.com/133135
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-09-04 14:52:30 +00:00
Ivan Kutuzov 5188c87c95 encoding/pem: fix for TestFuzz, PEM type should not contain a colon
Fixes #22238

Change-Id: I8184f789bd4120f3e71c9374c7c2fcbfa95935bf
Reviewed-on: https://go-review.googlesource.com/132635
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-09-03 15:04:25 +00:00
Taesu Pyo 21e85c293d encoding/json: fix UnmarshalTypeError without field and struct values
Fixes #26444
Fixes #27275

Change-Id: I9e8cbff79f7643ca8964c572c1a98172b6831730
GitHub-Last-Rev: 7eea2158b6
GitHub-Pull-Request: golang/go#26719
Reviewed-on: https://go-review.googlesource.com/126897
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-08-28 16:00:33 +00:00
Daniel Martí 969b9d8127 encoding/json: fix handling of nil anonymous structs
Given the following types:

	type S2 struct{ Field string }
	type S  struct{ *S2 }

Marshalling a value of type T1 should result in "{}", as there's no way
to access any value of T2.Field. This is how Go 1.10 and earlier
versions behave.

However, in the recent refactor golang.org/cl/125417 I broke this logic.
When the encoder found an anonymous struct pointer field that was nil,
it no longer skipped the embedded fields underneath it. This can be seen
in the added test:

	--- FAIL: TestAnonymousFields/EmbeddedFieldBehindNilPointer (0.00s)
	    encode_test.go:430: Marshal() = "{\"Field\":\"\\u003c*json.S2 Value\\u003e\"}", want "{}"

The human error was a misplaced label, meaning we weren't actually
skipping the right loop iteration. Fix that.

Change-Id: Iba8a4a77d358dac73dcba4018498fe4f81afa263
Reviewed-on: https://go-review.googlesource.com/131376
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-26 17:12:02 +00:00
Daniel Martí 21af0c1699 encoding/json: get rid of the stream_test.go TODOs
TestRawMessage now passes without the need for the RawMessage field to
be a pointer. The TODO dates all the way back to 2010, so I presume the
issue has since been fixed.

TestNullRawMessage tested the decoding of a JSON null into a
*RawMessage. The existing behavior was correct, but for the sake of
completeness a non-pointer RawMessage field has been added too. The
non-pointer field behaves differently, as one can read in the docs:

	To unmarshal JSON into a value implementing the Unmarshaler
	interface, Unmarshal calls that value's UnmarshalJSON method,
	including when the input is a JSON null.

Change-Id: Iabaed75d4ed10ea427d135ee1b80c6e6b83b2e6e
Reviewed-on: https://go-review.googlesource.com/131377
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-26 17:09:27 +00:00
Daniel Martí c21ba224ec encoding/json: remove a branch in the structEncoder loop
Encoders like map and array can use the much cheaper "i > 0" check to
see if we're not writing the first element. However, since struct fields
support omitempty, we need to keep track of that separately.

This is much more expensive - after calling the field encoder itself,
and retrieving the field via reflection, this branch was the third most
expensive piece of this field loop.

Instead, hoist the branch logic outside of the loop. The code doesn't
get much more complex, since we just delay the writing of each byte
until the next iteration. Yet the performance improvement is noticeable,
even when the struct types in CodeEncoder only have 2 and 7 fields,
respectively.

name           old time/op    new time/op    delta
CodeEncoder-4    5.39ms ± 0%    5.31ms ± 0%  -1.37%  (p=0.010 n=4+6)

name           old speed      new speed      delta
CodeEncoder-4   360MB/s ± 0%   365MB/s ± 0%  +1.39%  (p=0.010 n=4+6)

Updates #5683.

Change-Id: I2662cf459e0dfd68e56fa52bc898a417e84266c2
Reviewed-on: https://go-review.googlesource.com/131401
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-08-25 23:59:43 +00:00
Daniel Martí 88f4bccec5 encoding/json: avoid some more pointer receivers
A few encoder struct types, such as map and slice, only encapsulate
other prepared encoder funcs. Using pointer receivers has no advantage,
and makes calling these methods slightly more expensive.

Not a huge performance win, but certainly an easy one. The struct types
used in the benchmark below contain one slice field and one pointer
field.

name           old time/op    new time/op    delta
CodeEncoder-4    5.48ms ± 0%    5.39ms ± 0%  -1.66%  (p=0.010 n=6+4)

name           old speed      new speed      delta
CodeEncoder-4   354MB/s ± 0%   360MB/s ± 0%  +1.69%  (p=0.010 n=6+4)

Updates #5683.

Change-Id: I9f78dbe07fcc6fbf19a6d96c22f5d6970db9eca4
Reviewed-on: https://go-review.googlesource.com/131400
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-08-25 23:58:26 +00:00
Daniel Martí 811b187a4f encoding/base64: slight decoding speed-up
First, use a dummy slice access on decode64 and decode32 to ensure that
there is a single bounds check for src.

Second, move the PutUint64/PutUint32 calls out of these functions,
meaning that they are simpler and smaller. This may also open the door
to inlineability in the future, but for now, they both go past the
budget.

While at it, get rid of the ilen and olen variables, which have no
impact whatsoever on performance. At least, not measurable by any of the
benchmarks.

name                 old time/op    new time/op    delta
DecodeString/2-4       54.3ns ± 1%    55.2ns ± 2%   +1.60%  (p=0.017 n=5+6)
DecodeString/4-4       66.6ns ± 1%    66.8ns ± 2%     ~     (p=0.903 n=6+6)
DecodeString/8-4       79.3ns ± 2%    79.6ns ± 1%     ~     (p=0.448 n=6+6)
DecodeString/64-4       300ns ± 1%     281ns ± 3%   -6.54%  (p=0.002 n=6+6)
DecodeString/8192-4    27.4µs ± 1%    23.7µs ± 2%  -13.47%  (p=0.002 n=6+6)

name                 old speed      new speed      delta
DecodeString/2-4     73.7MB/s ± 1%  72.5MB/s ± 2%   -1.55%  (p=0.026 n=5+6)
DecodeString/4-4      120MB/s ± 1%   120MB/s ± 2%     ~     (p=0.851 n=6+6)
DecodeString/8-4      151MB/s ± 2%   151MB/s ± 1%     ~     (p=0.485 n=6+6)
DecodeString/64-4     292MB/s ± 1%   313MB/s ± 3%   +7.03%  (p=0.002 n=6+6)
DecodeString/8192-4   399MB/s ± 1%   461MB/s ± 2%  +15.58%  (p=0.002 n=6+6)

For #19636.

Change-Id: I0dfbdafa2a41dc4c582f63aef94b90b8e473731c
Reviewed-on: https://go-review.googlesource.com/113776
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-08-22 19:08:03 +00:00
Daniel Martí 9d1540b77c encoding/json: simplify some pieces of the encoder
Some WriteByte('\\') calls can be deduplicated.

fillField is used in two occasions, but it is unnecessary when adding
fields to the "next" stack, as those aren't used for the final encoding.
Inline the func with its only remaining call.

Finally, unindent a default-if block.

The performance of the encoder is unaffected:

name           old time/op    new time/op    delta
CodeEncoder-4    6.65ms ± 1%    6.65ms ± 0%   ~     (p=0.662 n=6+5)

Change-Id: Ie55baeab89abad9b9f13e9f6ca886a670c30dba9
Reviewed-on: https://go-review.googlesource.com/122461
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-22 16:14:05 +00:00
Daniel Martí 75e7e05aee encoding/json: inline fieldByIndex
This function was only used in a single place - in the field encoding
loop within the struct encoder.

Inlining the function call manually lets us get rid of the call
overhead. But most importantly, it lets us simplify the logic afterward.
We no longer need to use reflect.Value{} and !fv.IsValid(), as we can
skip the field immediately.

The two factors combined (mostly just the latter) give a moderate speed
improvement to this hot loop.

name           old time/op    new time/op    delta
CodeEncoder-4    6.01ms ± 1%    5.91ms ± 1%  -1.66%  (p=0.002 n=6+6)

name           old speed      new speed      delta
CodeEncoder-4   323MB/s ± 1%   328MB/s ± 1%  +1.69%  (p=0.002 n=6+6)

Updates #5683.

Change-Id: I12757c325a68abb2856026cf719c122612a1f38e
Reviewed-on: https://go-review.googlesource.com/125417
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-08-22 15:55:40 +00:00
Daniel Martí 8148726676 encoding/json: simplify the structEncoder type
structEncoder had two slices - the list of fields, and a list containing
the encoder for each field. structEncoder.encode then looped over the
fields, and indexed into the second slice to grab the field encoder.

However, this makes it very hard for the compiler to be able to prove
that the two slices always have the same length, and that the index
expression doesn't need a bounds check.

Merge the two slices into one to completely remove the need for bounds
checks in the hot loop.

While at it, don't copy the field elements when ranging, which greatly
speeds up the hot loop in structEncoder.

name           old time/op    new time/op    delta
CodeEncoder-4    6.18ms ± 0%    5.56ms ± 0%  -10.08%  (p=0.002 n=6+6)

name           old speed      new speed      delta
CodeEncoder-4   314MB/s ± 0%   349MB/s ± 0%  +11.21%  (p=0.002 n=6+6)

name           old alloc/op   new alloc/op   delta
CodeEncoder-4    93.2kB ± 0%    62.1kB ± 0%  -33.33%  (p=0.002 n=6+6)

Updates #5683.

Change-Id: I0dd47783530f439b125e084aede09dda172eb1e8
Reviewed-on: https://go-review.googlesource.com/125416
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-22 15:08:21 +00:00
Daniel Martí 6d4787aff2 encoding/json: various minor decoder speed-ups
Reuse v.Type() and cachedTypeFields(t) when decoding maps and structs.

Always use the same data slices when in hot loops, to ensure that the
compiler generates good code. "for i < len(data) { use(d.data[i]) }"
makes it harder for the compiler.

Finally, do other minor clean-ups, such as deduplicating switch cases,
and using a switch instead of three chained ifs.

The decoder sees a noticeable speed-up, in particular when decoding
structs.

name           old time/op    new time/op    delta
CodeDecoder-4    29.8ms ± 1%    27.5ms ± 0%  -7.83%  (p=0.002 n=6+6)

name           old speed      new speed      delta
CodeDecoder-4  65.0MB/s ± 1%  70.6MB/s ± 0%  +8.49%  (p=0.002 n=6+6)

Updates #5683.

Change-Id: I9d751e22502221962da696e48996ffdeb777277d
Reviewed-on: https://go-review.googlesource.com/122468
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-21 19:00:08 +00:00
Daniel Martí 9a2a34e1c1 encoding/json: defer error context work until necessary
Calling Name on a reflect.Type is somewhat expensive, as it involves a
number of nested calls and string handling.

This cost was showing up when decoding structs, as we were calling it to
set up an error context.

We can avoid the extra work unless we do encounter an error, which makes
decoding via struct types faster.

name           old time/op    new time/op    delta
CodeDecoder-4    31.0ms ± 1%    29.9ms ± 1%  -3.69%  (p=0.002 n=6+6)

name           old speed      new speed      delta
CodeDecoder-4  62.6MB/s ± 1%  65.0MB/s ± 1%  +3.83%  (p=0.002 n=6+6)

Updates #5683.

Change-Id: I48a3a85ef0ba96f524b7c3e9096cb2c4589e077a
Reviewed-on: https://go-review.googlesource.com/122467
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-21 18:18:59 +00:00
Daniel Martí 30d3ebe367 encoding/json: remove alloc when encoding short byte slices
If the encoded bytes fit in the bootstrap array encodeState.scratch, use
that instead of allocating a new byte slice.

Also tweaked the Encoding vs Encoder heuristic to use the length of the
encoded bytes, not the length of the input bytes. Encoding is used for
allocations of up to 1024 bytes, as we measured 2048 to be the point
where it no longer provides a noticeable advantage.

Also added some benchmarks. Only the first case changes in behavior.

name                 old time/op    new time/op    delta
MarshalBytes/32-4       420ns ± 1%     383ns ± 1%   -8.69%  (p=0.002 n=6+6)
MarshalBytes/256-4      913ns ± 1%     915ns ± 0%     ~     (p=0.580 n=5+6)
MarshalBytes/4096-4    7.72µs ± 0%    7.74µs ± 0%     ~     (p=0.340 n=5+6)

name                 old alloc/op   new alloc/op   delta
MarshalBytes/32-4        112B ± 0%       64B ± 0%  -42.86%  (p=0.002 n=6+6)
MarshalBytes/256-4       736B ± 0%      736B ± 0%     ~     (all equal)
MarshalBytes/4096-4    7.30kB ± 0%    7.30kB ± 0%     ~     (all equal)

name                 old allocs/op  new allocs/op  delta
MarshalBytes/32-4        2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.002 n=6+6)
MarshalBytes/256-4       2.00 ± 0%      2.00 ± 0%     ~     (all equal)
MarshalBytes/4096-4      2.00 ± 0%      2.00 ± 0%     ~     (all equal)

Updates #5683.

Change-Id: I5fa55c27bd7728338d770ae7c0756885ba9a5724
Reviewed-on: https://go-review.googlesource.com/122462
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-21 09:05:51 +00:00
Daniel Martí 2d3599e57d encoding/json: encode struct field names ahead of time
Struct field names are static, so we can run HTMLEscape on them when
building each struct type encoder. Then, when running the struct
encoder, we can select either the original or the escaped field name to
write directly.

When the encoder is not escaping HTML, using the original string works
because neither Go struct field names nor JSON tags allow any characters
that would need to be escaped, like '"', '\\', or '\n'.

When the encoder is escaping HTML, the only difference is that '<', '>',
and '&' are allowed via JSON struct field tags, hence why we use
HTMLEscape to properly escape them.

All of the above lets us encode field names with a simple if/else and
WriteString calls, which are considerably simpler and faster than
encoding an arbitrary string.

While at it, also include the quotes and colon in these strings, to
avoid three WriteByte calls in the loop hot path.

Also added a few tests, to ensure that the behavior in these edge cases
is not broken. The output of the tests is the same if this optimization
is reverted.

name           old time/op    new time/op    delta
CodeEncoder-4    7.12ms ± 0%    6.14ms ± 0%  -13.85%  (p=0.004 n=6+5)

name           old speed      new speed      delta
CodeEncoder-4   272MB/s ± 0%   316MB/s ± 0%  +16.08%  (p=0.004 n=6+5)

name           old alloc/op   new alloc/op   delta
CodeEncoder-4    91.9kB ± 0%    93.2kB ± 0%   +1.43%  (p=0.002 n=6+6)

name           old allocs/op  new allocs/op  delta
CodeEncoder-4      0.00           0.00          ~     (all equal)

Updates #5683.

Change-Id: I6f6a340d0de4670799ce38cf95b2092822d2e3ef
Reviewed-on: https://go-review.googlesource.com/122460
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-21 09:05:48 +00:00
Tim Cooper dc272a4393 encoding/json: call reflect.TypeOf with nil pointers rather than allocating
Updates #26775

Change-Id: I83c9eeda59769d2f35e0cc98f3a8579861d5978b
Reviewed-on: https://go-review.googlesource.com/119715
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-08-21 02:38:02 +00:00
Philip Børgesen 97c7e0e0ad encoding/json: eliminate superfluous space in Decoder.Token error messages
The existing Decoder.tokenError implementation creates its error messages by
concatenating "invalid character " + quoteChar(c) + " " + context. All context
values however already start with a space leading to error messages containing
two spaces.

This change removes " " from the concatenation expression.

Fixes #26587

Change-Id: I93d14319396636b2a40d55053bda88c98e94a81a
GitHub-Last-Rev: 6db7e1991b
GitHub-Pull-Request: golang/go#26588
Reviewed-on: https://go-review.googlesource.com/125775
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-08-21 02:26:17 +00:00
Tim Cooper 352583ff77 encoding/hex: pre-allocate Dump buffer
name          old time/op    new time/op    delta
Dump/256-4      7.76µs ± 2%    5.82µs ± 2%  -24.91%  (p=0.008 n=5+5)
Dump/1024-4     28.4µs ± 2%    22.6µs ± 3%  -20.58%  (p=0.008 n=5+5)
Dump/4096-4      112µs ± 2%      88µs ± 0%  -20.80%  (p=0.016 n=5+4)
Dump/16384-4     444µs ± 3%     361µs ± 7%  -18.73%  (p=0.008 n=5+5)

name          old alloc/op   new alloc/op   delta
Dump/256-4      4.00kB ± 0%    1.39kB ± 0%  -65.20%  (p=0.008 n=5+5)
Dump/1024-4     16.2kB ± 0%     5.5kB ± 0%  -66.04%  (p=0.008 n=5+5)
Dump/4096-4     63.9kB ± 0%    20.6kB ± 0%  -67.78%  (p=0.008 n=5+5)
Dump/16384-4     265kB ± 0%      82kB ± 0%  -69.00%  (p=0.008 n=5+5)

name          old allocs/op  new allocs/op  delta
Dump/256-4        7.00 ± 0%      3.00 ± 0%  -57.14%  (p=0.008 n=5+5)
Dump/1024-4       9.00 ± 0%      3.00 ± 0%  -66.67%  (p=0.008 n=5+5)
Dump/4096-4       11.0 ± 0%       3.0 ± 0%  -72.73%  (p=0.008 n=5+5)
Dump/16384-4      13.0 ± 0%       3.0 ± 0%  -76.92%  (p=0.008 n=5+5)

Change-Id: I0a0d6de315b979142b05e333880da8a5e52b12ef
Reviewed-on: https://go-review.googlesource.com/116495
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-21 02:10:52 +00:00
Brad Fitzpatrick 2069543bf1 encoding/xml: document HTMLAutoClose and HTMLEntity more
They didn't even have public types, which made them pretty mysterious.

Give them types and reference the Decoder, which uses them.

Also, refer them qualified by their package name in the examples, as
we usually do in example*.go files, which usually use package foo_test
specifically so we can show the package names along with the symbols.

Change-Id: I50ebbbf43778c1627bfa526f8824f52c7953454f
Reviewed-on: https://go-review.googlesource.com/127663
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-08-02 20:47:47 +00:00
Brad Fitzpatrick 2f46dfb271 encoding/xml: remove some primordial semicolons
Change-Id: I23e5d87648a4091fb4f6616bf80aa6c800974900
Reviewed-on: https://go-review.googlesource.com/127662
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2018-08-02 20:40:06 +00:00
Ian Lance Taylor f297d165c9 encoding/gob: correct issue number typo in comment
Change-Id: I3ac25cf1770b5ac0d36690c37615b3badd27463d
Reviewed-on: https://go-review.googlesource.com/118455
Reviewed-by: Rob Pike <r@golang.org>
2018-06-13 05:33:59 +00:00
Niek Sanders b768d82d8e encoding/base32: eliminate alphabet bounds check
name              old time/op   new time/op   delta
EncodeToString-4   35.5µs ± 7%   33.3µs ± 6%  -6.27%   (p=0.008 n=10+9)
DecodeString-4      120µs ± 7%    113µs ± 8%  -5.88%  (p=0.011 n=10+10)

name              old speed     new speed     delta
EncodeToString-4  231MB/s ± 8%  247MB/s ± 5%  +6.55%   (p=0.008 n=10+9)
DecodeString-4    109MB/s ± 7%  116MB/s ± 8%  +6.27%  (p=0.011 n=10+10)

Change-Id: I60bf962464179e35b1711617adbc45a822eaece5
Reviewed-on: https://go-review.googlesource.com/45876
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-06-12 22:51:30 +00:00
Tobias Klauser db4675f2de encoding/asn1: gofmt
CL 113837 introduced some changes which were not properly gofmt'ed, fix them.

Change-Id: I89329063f9c468238051e45380d752e66efdb939
Reviewed-on: https://go-review.googlesource.com/116895
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-06-07 15:01:06 +00:00
Tim Cooper 161874da2a all: update comment URLs from HTTP to HTTPS, where possible
Each URL was manually verified to ensure it did not serve up incorrect
content.

Change-Id: I4dc846227af95a73ee9a3074d0c379ff0fa955df
Reviewed-on: https://go-review.googlesource.com/115798
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
2018-06-01 21:52:00 +00:00
Josh Bleecher Snyder a34e6650c0 encoding/hex: improve Decode and DecodeString docs
Simplify the wording of both.

Make the DecodeString docs more accurate:
DecodeString returns a slice, not a string.

Change-Id: Iba7003f55fb0a37aafcbeee59a30492c0f68aa4e
Reviewed-on: https://go-review.googlesource.com/115615
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-05-31 19:49:01 +00:00
Constantin Konstantinidis 6b4828a206 encoding/asn1: fix returned type for an Object Identifier
Unmarshal/Marshal/Unmarshal was not idempotent as the Object Identifier
type was not returned through the interface. The limit case OID = 0
returns an error. The zero OID is 0.0

A test is fixed to use the Object Identifier type.
Other related test are added.

Fixes #11130

Change-Id: I15483a3126066c9b99cf5bd9c4b0cc15ec1d61ca
Reviewed-on: https://go-review.googlesource.com/113837
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
2018-05-31 15:24:46 +00:00
dchenk faa69b906f encoding/base32: remove redundant conditional
Immediately following the conditional block removed here is a loop
which checks exactly what the conditional already checked, so the
entire conditional is redundant.

Change-Id: I892fd9f2364d87e2c1cacb0407531daec6643183
Reviewed-on: https://go-review.googlesource.com/114000
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-05-22 17:12:05 +00:00
HAMANO Tsukasa f2239d3957 encoding/asn1: allow Marshaling and Unmarshaling private tag class
ASN.1 has an private class, but current implementation does not support it.

Change-Id: I3ebf07a048831869572f75223cb17d4c115caef7
GitHub-Last-Rev: b3c69ad091
GitHub-Pull-Request: golang/go#25195
Reviewed-on: https://go-review.googlesource.com/110561
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
2018-05-17 18:06:38 +00:00
Gustav Westling c3d10e64b9 encoding/base32: handle NoPadding in NewDecoder
This change adds functionality to properly handle NoPadding in NewDecoder.

Removes the following expectations when using NoPadding:

* the input message length is a multiple of 8
* the input message length is 0, or longer than 7 characters

Fixes #25332

Change-Id: I7c38160df23f7e8da4f85a5629530016e7bf71f3
GitHub-Last-Rev: 68ab8d2291
GitHub-Pull-Request: golang/go#25394
Reviewed-on: https://go-review.googlesource.com/113215
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-05-16 21:15:03 +00:00
Gustav Westling 10529a01fd encoding/base32: handle NoPadding when using buffered encoding in Close
This changes makes encoder.Close aware of how many bytes to write if there
is any data left in the buffer.

Fixes #25295

Change-Id: I4138891359935509cb561c453b8059ba2b9e576b
GitHub-Last-Rev: f374096d2f
GitHub-Pull-Request: golang/go#25316
Reviewed-on: https://go-review.googlesource.com/112515
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-05-09 22:10:32 +00:00
Gustav Westling 0f2d4d0008 encoding/base32: handle surplus padding consistently
This changes decoder.Read to always return io.ErrUnexpectedEOF if the input
contains surplus padding or unexpected content. Previously the error could
be io.EOF or io.ErrUnexpectedEOF depending on how the input was chunked.

Fixes #25296

Change-Id: I07c36c35e6c83e795c3991bfe45647a35aa58aa4
GitHub-Last-Rev: 818dfda90b
GitHub-Pull-Request: golang/go#25319
Reviewed-on: https://go-review.googlesource.com/112516
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-05-09 22:09:56 +00:00
Artyom Pervukhin 4410934cba encoding/xml: fix valid character range
Section 2.2 of the referenced spec http://www.xml.com/axml/testaxml.htm
defines 0xD7FF as a (sub)range boundary, not 0xDF77.

Fixes #25172

Change-Id: Ic5a3328cd46ef6474b8e93c4a343dcfba0e6511f
Reviewed-on: https://go-review.googlesource.com/109495
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-05-09 17:31:08 +00:00
Joe Kyo 5188b4dea0 encoding/binary: returns length of bool slice in intDataSize
intDataSize should return length of bool slice, so functions
Read and Write can use the fast path to process bool slice.

Change-Id: I8cd275e3ffea82024850662d86caca64bd91bf70
Reviewed-on: https://go-review.googlesource.com/112135
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
2018-05-08 14:48:50 +00:00
Richard Musiol e3c684777a all: skip unsupported tests for js/wasm
The general policy for the current state of js/wasm is that it only
has to support tests that are also supported by nacl.

The test nilptr3.go makes assumptions about which nil checks can be
removed. Since WebAssembly does not signal on reading a null pointer,
all nil checks have to be explicit.

Updates #18892

Change-Id: I06a687860b8d22ae26b1c391499c0f5183e4c485
Reviewed-on: https://go-review.googlesource.com/110096
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-04-30 19:39:18 +00:00
Russ Cox 50a583966a encoding/base64: fix format error
Found by pending CL to make cmd/vet auto-detect printf wrappers.

Change-Id: I2ad06647b7b41cf68859820a60eeac2e689ca2e6
Reviewed-on: https://go-review.googlesource.com/109344
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-04-25 20:22:16 +00:00
tengufromsky c2a53b1b82 encoding/json: remove unnecessary if conditions
Fixes gosimple warning "if err != nil { return err };
return nil' can be simplified to 'return err"

Change-Id: Ife7f78a3a76ab7802b5561d1afec536e103b504a
Reviewed-on: https://go-review.googlesource.com/108275
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-04-20 01:41:41 +00:00
Michael Fraenkel fc21598931 encoding/json: simplify dominantField
Fixes #18037

Change-Id: I20e27bcc013b00b726eb348daf5ca86b138ddcc2
Reviewed-on: https://go-review.googlesource.com/107598
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-04-17 23:04:19 +00:00
Tim Cooper 9db1dd074d encoding/hex: fix Dumper not always closing on Close call
Updates #23574

Change-Id: I1b87390679e0817a2f6e4e5938994ea32df87bd7
Reviewed-on: https://go-review.googlesource.com/107596
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-04-17 20:14:55 +00:00