Commit Graph

335 Commits

Author SHA1 Message Date
Josh Bleecher Snyder b85433975a encoding/binary: use an offset instead of slicing
While running make.bash, over 5% of all pointer writes
come from encoding/binary doing struct reads.

This change replaces slicing during such reads with an offset.
This avoids updating the slice pointer with every
struct field read or write.

This has no impact when the write barrier is off.
Running the benchmarks with GOGC=1, however,
shows significant improvement:

name          old time/op    new time/op    delta
ReadStruct-8    13.2µs ± 6%    10.1µs ± 5%  -23.24%  (p=0.000 n=10+10)

name          old speed      new speed      delta
ReadStruct-8  5.69MB/s ± 6%  7.40MB/s ± 5%  +30.18%  (p=0.000 n=10+10)

Change-Id: I22904263196bfeddc38abe8989428e263aee5253
Reviewed-on: https://go-review.googlesource.com/98757
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-03-06 18:59:03 +00:00
Pascal S. de Kloe 74a92b8e8d encoding/json: apply conventional error handling in decoder
name                            old time/op    new time/op    delta
CodeEncoder-12                    1.89ms ± 1%    1.91ms ± 0%   +1.16%  (p=0.000 n=20+19)
CodeMarshal-12                    2.09ms ± 1%    2.12ms ± 0%   +1.63%  (p=0.000 n=17+18)
CodeDecoder-12                    8.43ms ± 1%    8.32ms ± 1%   -1.35%  (p=0.000 n=18+20)
UnicodeDecoder-12                  399ns ± 0%     339ns ± 0%  -15.00%  (p=0.000 n=20+19)
DecoderStream-12                   281ns ± 1%     231ns ± 0%  -17.91%  (p=0.000 n=20+16)
CodeUnmarshal-12                  9.35ms ± 2%    9.15ms ± 2%   -2.11%  (p=0.000 n=20+20)
CodeUnmarshalReuse-12             8.41ms ± 2%    8.29ms ± 2%   -1.34%  (p=0.000 n=20+20)
UnmarshalString-12                81.2ns ± 2%    74.0ns ± 4%   -8.89%  (p=0.000 n=20+20)
UnmarshalFloat64-12               71.1ns ± 2%    64.3ns ± 1%   -9.60%  (p=0.000 n=20+19)
UnmarshalInt64-12                 60.6ns ± 2%    53.2ns ± 0%  -12.28%  (p=0.000 n=18+18)
Issue10335-12                     96.9ns ± 0%    87.7ns ± 1%   -9.52%  (p=0.000 n=17+20)
Unmapped-12                        247ns ± 4%     231ns ± 3%   -6.34%  (p=0.000 n=20+20)
TypeFieldsCache/MissTypes1-12     11.1µs ± 0%    11.1µs ± 0%     ~     (p=0.376 n=19+20)
TypeFieldsCache/MissTypes10-12    33.9µs ± 0%    33.8µs ± 0%   -0.32%  (p=0.000 n=18+9)

name                            old speed      new speed      delta
CodeEncoder-12                  1.03GB/s ± 1%  1.01GB/s ± 0%   -1.15%  (p=0.000 n=20+19)
CodeMarshal-12                   930MB/s ± 1%   915MB/s ± 0%   -1.60%  (p=0.000 n=17+18)
CodeDecoder-12                   230MB/s ± 1%   233MB/s ± 1%   +1.37%  (p=0.000 n=18+20)
UnicodeDecoder-12               35.0MB/s ± 0%  41.2MB/s ± 0%  +17.60%  (p=0.000 n=20+19)
CodeUnmarshal-12                 208MB/s ± 2%   212MB/s ± 2%   +2.16%  (p=0.000 n=20+20)

name                            old alloc/op   new alloc/op   delta
Issue10335-12                       184B ± 0%      184B ± 0%     ~     (all equal)
Unmapped-12                         216B ± 0%      216B ± 0%     ~     (all equal)

name                            old allocs/op  new allocs/op  delta
Issue10335-12                       3.00 ± 0%      3.00 ± 0%     ~     (all equal)
Unmapped-12                         4.00 ± 0%      4.00 ± 0%     ~     (all equal)

Change-Id: I4b1a87a205da2ef9a572f86f85bc833653c61570
Reviewed-on: https://go-review.googlesource.com/98440
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-03-03 17:16:47 +00:00
Joe Tsai f0756ca2ea encoding/json: use sync.Map for field cache
The previous type cache is quadratic in time in the situation where
new types are continually encountered. Now that it is possible to dynamically
create new types with the reflect package, this can cause json to
perform very poorly.

Switch to sync.Map which does well when the cache has hit steady state,
but also handles occasional updates in better than quadratic time.

benchmark                                     old ns/op      new ns/op     delta
BenchmarkTypeFieldsCache/MissTypes1-8         14817          16202         +9.35%
BenchmarkTypeFieldsCache/MissTypes10-8        70926          69144         -2.51%
BenchmarkTypeFieldsCache/MissTypes100-8       976467         208973        -78.60%
BenchmarkTypeFieldsCache/MissTypes1000-8      79520162       1750371       -97.80%
BenchmarkTypeFieldsCache/MissTypes10000-8     6873625837     16847806      -99.75%
BenchmarkTypeFieldsCache/HitTypes1000-8       7.51           8.80          +17.18%
BenchmarkTypeFieldsCache/HitTypes10000-8      7.58           8.68          +14.51%

The old implementation takes 12 minutes just to build a cache of size 1e5
due to the quadratic behavior. I did not bother benchmark sizes above that.

Change-Id: I5e6facc1eb8e1b80e5ca285e4dd2cc8815618dad
Reviewed-on: https://go-review.googlesource.com/76850
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-03-03 00:08:09 +00:00
Pascal S. de Kloe 5d11838654 encoding/json: read ahead after value consumption
Eliminates the need for an extra scanner, read undo and some other tricks.

name                    old time/op    new time/op    delta
CodeEncoder-12            1.92ms ± 0%    1.91ms ± 1%   -0.65%  (p=0.000 n=17+20)
CodeMarshal-12            2.13ms ± 2%    2.12ms ± 1%   -0.49%  (p=0.038 n=18+17)
CodeDecoder-12            8.55ms ± 2%    8.49ms ± 1%     ~     (p=0.119 n=20+18)
UnicodeDecoder-12          411ns ± 0%     422ns ± 0%   +2.77%  (p=0.000 n=19+15)
DecoderStream-12           320ns ± 1%     307ns ± 1%   -3.80%  (p=0.000 n=18+20)
CodeUnmarshal-12          9.65ms ± 3%    9.58ms ± 3%     ~     (p=0.157 n=20+20)
CodeUnmarshalReuse-12     8.54ms ± 3%    8.56ms ± 2%     ~     (p=0.602 n=20+20)
UnmarshalString-12         110ns ± 1%      87ns ± 2%  -21.53%  (p=0.000 n=16+20)
UnmarshalFloat64-12        101ns ± 1%      77ns ± 2%  -23.08%  (p=0.000 n=19+20)
UnmarshalInt64-12         94.5ns ± 2%    68.4ns ± 1%  -27.60%  (p=0.000 n=20+20)
Issue10335-12              128ns ± 1%     100ns ± 1%  -21.89%  (p=0.000 n=19+18)
Unmapped-12                427ns ± 3%     247ns ± 4%  -42.17%  (p=0.000 n=20+20)
NumberIsValid-12          23.0ns ± 0%    21.7ns ± 0%   -5.73%  (p=0.000 n=20+20)
NumberIsValidRegexp-12     641ns ± 0%     642ns ± 0%   +0.15%  (p=0.003 n=19+19)
EncoderEncode-12          56.9ns ± 0%    55.0ns ± 1%   -3.32%  (p=0.012 n=2+17)

name                    old speed      new speed      delta
CodeEncoder-12          1.01GB/s ± 1%  1.02GB/s ± 1%   +0.71%  (p=0.000 n=18+20)
CodeMarshal-12           913MB/s ± 2%   917MB/s ± 1%   +0.49%  (p=0.038 n=18+17)
CodeDecoder-12           227MB/s ± 2%   229MB/s ± 1%     ~     (p=0.110 n=20+18)
UnicodeDecoder-12       34.1MB/s ± 0%  33.1MB/s ± 0%   -2.73%  (p=0.000 n=19+19)
CodeUnmarshal-12         201MB/s ± 3%   203MB/s ± 3%     ~     (p=0.151 n=20+20)

name                    old alloc/op   new alloc/op   delta
Issue10335-12               320B ± 0%      184B ± 0%  -42.50%  (p=0.000 n=20+20)
Unmapped-12                 568B ± 0%      216B ± 0%  -61.97%  (p=0.000 n=20+20)
EncoderEncode-12           0.00B          0.00B          ~     (all equal)

name                    old allocs/op  new allocs/op  delta
Issue10335-12               4.00 ± 0%      3.00 ± 0%  -25.00%  (p=0.000 n=20+20)
Unmapped-12                 18.0 ± 0%       4.0 ± 0%  -77.78%  (p=0.000 n=20+20)
EncoderEncode-12            0.00           0.00          ~     (all equal)

Fixes #17914
Updates #20693
Updates #10335

Change-Id: I0459a52febb8b79c9a2991e69ed2614cf8740429
Reviewed-on: https://go-review.googlesource.com/47152
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-03-01 19:03:04 +00:00
Joe Tsai 4338518da8 encoding/json: avoid assuming side-effect free reflect.Value.Addr().Elem()
Consider the following:
	type child struct{ Field string }
	type parent struct{ child }

	p := new(parent)
	v := reflect.ValueOf(p).Elem().Field(0)
	v.Field(0).SetString("hello")           // v.Field = "hello"
	v = v.Addr().Elem()                     // v = *(&v)
	v.Field(0).SetString("goodbye")         // v.Field = "goodbye"

It would appear that v.Addr().Elem() should have the same value, and
that it would be safe to set "goodbye".
However, after CL 66331, any interspersed calls between Field calls
causes the RO flag to be set.
Thus, setting to "goodbye" actually causes a panic.

That CL affects decodeState.indirect which assumes that back-to-back
Value.Addr().Elem() is side-effect free. We fix that logic to keep
track of the Addr() and Elem() calls and set v back to the original
after a full round-trip has occured.

Fixes #24152
Updates #24153

Change-Id: Ie50f8fe963f00cef8515d89d1d5cbc43b76d9f9c
Reviewed-on: https://go-review.googlesource.com/97796
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-03-01 00:16:20 +00:00
Ryuma Yoshida 8fc25b531b all: remove duplicate word "the"
Change-Id: Ia5908e94a6bd362099ca3c63f6ffb7e94457131d
GitHub-Last-Rev: 545a40571a
GitHub-Pull-Request: golang/go#23942
Reviewed-on: https://go-review.googlesource.com/95435
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-02-20 16:45:55 +00:00
Alberto Donizetti 252ee90971 encoding/xml: simplify slice-growing logic in rawToken
It appears that old code (from 2009) in xml.(*Decoder).rawToken
replicates append's slice-growing functionality by allocating a new,
bigger backing array and then calling copy.

Simplifying the code by replacing it with a single append call does
not seem to hurt performance:

name         old time/op    new time/op    delta
Marshal-4      11.2µs ± 1%    11.3µs ±10%    ~     (p=0.069 n=19+17)
Unmarshal-4    28.6µs ± 1%    28.4µs ± 1%  -0.60%  (p=0.000 n=20+18)

name         old alloc/op   new alloc/op   delta
Marshal-4      5.78kB ± 0%    5.78kB ± 0%    ~     (all equal)
Unmarshal-4    8.61kB ± 0%    8.27kB ± 0%  -3.90%  (p=0.000 n=20+20)

name         old allocs/op  new allocs/op  delta
Marshal-4        23.0 ± 0%      23.0 ± 0%    ~     (all equal)
Unmarshal-4       189 ± 0%       190 ± 0%  +0.53%  (p=0.000 n=20+20)

Change-Id: Ie580d1216a44760e611e63dee2c339af5465aea5
Reviewed-on: https://go-review.googlesource.com/86655
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-02-19 23:42:57 +00:00
Joe Tsai 91a6a2a30f encoding/json: make error capture logic in recover more type safe
Rather than only ignoring runtime.Error panics, which are a very
narrow set of possible panic values, switch it such that the json
package only captures panic values that have been properly wrapped
in a jsonError struct. This ensures that only intentional panics
originating from the json package are captured.

Fixes #23012

Change-Id: I5e85200259edd2abb1b0512ce6cc288849151a6d
Reviewed-on: https://go-review.googlesource.com/94019
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-02-14 21:34:26 +00:00
Tim Cooper 0519126a3f encoding/hex: fix potential incorrect Dumper output when Close is called multiple times
Fixes #23574

Change-Id: I69573de47daa6fd53cc99a78c0c4b867460242e3
Reviewed-on: https://go-review.googlesource.com/90275
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
2018-02-14 03:25:25 +00:00
Ian Lance Taylor 1a9f27d503 encoding/gob: avoid race on idToType
Fixes #23328

Change-Id: Ie4864d7f388d363860318fe41431d8a9719e9a75
Reviewed-on: https://go-review.googlesource.com/86075
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-01-04 02:17:33 +00:00
Igor Vashyst 38c561cb2c encoding/xml: remove duplicate test of element presence
Change-Id: If0d9ff107fc6bbdf0231cd48abc23a44816bfe77
Reviewed-on: https://go-review.googlesource.com/85755
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-12-31 02:09:26 +00:00
Ian Lance Taylor 89d7a2fbda encoding/xml: don't crash on invalid XMLName tag
Fixes #20953

Change-Id: Ia30a6e0e335c1f738e1359500e09057b5981f1c7
Reviewed-on: https://go-review.googlesource.com/82397
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-07 05:09:01 +00:00
Joe Tsai 70f441bc49 encoding/json: error when trying to set an embedded pointer to unexported struct types
This CL reverts CL 76851 and takes a different approach to #21357.
The changes in encode.go and encode_test.go are reverts that
rolls back the changed behavior in CL 76851 where
embedded pointers to unexported struct types were
unilaterally ignored in both marshal and unmarshal.

Instead, these fields are handled as before with the exception that
it returns an error when Unmarshal is unable to set an unexported field.
The behavior of Marshal is now unchanged with regards to #21357.

This policy maintains the greatest degree of backwards compatibility
and avoids silently discarding data the user may have expected to be present.

Fixes #21357

Change-Id: I7dc753280c99f786ac51acf7e6c0246618c8b2b1
Reviewed-on: https://go-review.googlesource.com/82135
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-12-06 19:27:26 +00:00
Russ Cox c3fa046f54 encoding/pem: change Encode, EncodeToMemory not to generate partial PEM blocks
Originally these routines could not fail except by
returning errors from the underlying writer.

Then we realized that header keys containing colons
needed to be rejected, and we started returning an error
from Encode. But that only happens after writing a
partial PEM block to the underlying writer, which is
unfortunate, but at least it was undocumented.

CL 77790 then documented this unfortunate behavior.

Instead of documenting unfortunate behavior, fix it.

Change-Id: Ic7467a576c4cecd16a99138571a1269cc4f96204
Reviewed-on: https://go-review.googlesource.com/82076
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-06 03:59:41 +00:00
Joe Tsai 0b3b5113c0 encoding/csv: truncate carriage returns at EOF
This fixes a regression where only CRLF was folded into LF at EOF.
Now, we also truncate trailing CR at EOF to preserve the old behavior.

Every one of the test cases added exactly matches the behavior
of Go1.9, even if the results are somewhat unexpected.

Fixes #22937

Change-Id: I1bc6550533163ae489ea77ec1e598163267b7eec
Reviewed-on: https://go-review.googlesource.com/81577
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-12-05 18:44:31 +00:00
christopher-henderson eb441e6d21 encoding/asn1: allow '&' in PrintableString fields
There are, unfortunately, intermediate CA ceritificates in circulation
that contain the invalid character '&' in some PrintableString fields,
notably Organization Name. This patch allows for ampersand
to be parsed as though it is valid in an ASN.1 PrintableString.

Fixes #22970

Change-Id: Ifab1a10bbff1cdac68e843c6b857ff1a031051aa
Reviewed-on: https://go-review.googlesource.com/81635
Reviewed-by: Adam Langley <agl@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Adam Langley <agl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-12-05 18:22:53 +00:00
Joe Tsai b53088a634 Revert "go/printer: forbid empty line before first comment in block"
This reverts commit 08f19bbde1.

Reason for revert:
The changed transformation takes effect on a larger set
of code snippets than expected.

For example, this:
    func foo() {

        // Comment
        bar()

    }
becomes:
    func foo() {
        // Comment
        bar()

    }

This is an unintended consequence.

Change-Id: Ifca88d6267dab8a8170791f7205124712bf8ace8
Reviewed-on: https://go-review.googlesource.com/81335
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Joe Tsai <joetsai@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-12-01 01:12:26 +00:00
rajender 671cf92c32 encoding/json: remove the word "text" in "JSON text" from package docs.
It was added in CL 79995. It is unnecessarily confusing.

Change-Id: Ib8ff35b9f71b54ff99d2d6e0534c7128e1f4345a
Reviewed-on: https://go-review.googlesource.com/80035
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-27 18:51:36 +00:00
rajender 0d26474606 encoding/json: update RFC number
Existing docs mention obsolete RFC 4627. Update it with current one,
https://tools.ietf.org/html/rfc7159.

Current implementation already adhere to RFC 7159.

Fixes #22888

Change-Id: I705ec1313f6f655b3bc41d2f847b30e479bf9b15
Reviewed-on: https://go-review.googlesource.com/79995
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-27 18:04:57 +00:00
Mansour Rahimi 41d6c89e1e encoding/asn1: support Unmarshaling NumericString
ASN.1 has an specific string type, called NumericString (tag 18). The
value of this type can be numeric characters (0-9) and space.

Fixes #22396

Change-Id: Ia6d81ab7faa311ff22759bf76862626974d3013e
Reviewed-on: https://go-review.googlesource.com/78655
Run-TryBot: Adam Langley <agl@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-25 17:08:32 +00:00
Michael Schurter 7eb7f8f5a7 encoding/json: reduce allocations by Decoder for \uXXXX
Manually convert hex escape sequence to rune instead of calling
strconv.ParseUint.

This inlines the unhex func from docs (and many other packages).

name              old time/op    new time/op    delta
UnicodeDecoder-4     468ns ± 1%     402ns ± 1%  -14.26%  (p=0.000
n=10+10)

name              old speed      new speed      delta
UnicodeDecoder-4  29.9MB/s ± 1%  34.8MB/s ± 1%  +16.59%  (p=0.000
n=10+10)

name              old alloc/op   new alloc/op   delta
UnicodeDecoder-4     44.0B ± 0%     36.0B ± 0%  -18.18%  (p=0.000
n=10+10)

name              old allocs/op  new allocs/op  delta
UnicodeDecoder-4      4.00 ± 0%      2.00 ± 0%  -50.00%  (p=0.000
n=10+10)

Fixes #20567

Change-Id: If350978d5bb98ff517485752184d02249f5d1f3a
Reviewed-on: https://go-review.googlesource.com/44738
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-11-22 03:36:29 +00:00
Russ Cox 1d47a14591 encoding/csv: restore Go 1.9 quoted \r\n handling in Reader
CL 52810 changed Reader to interpret a quoted \r\n as a raw \r\n
when reading fields. This seems likely to break existing users, and
discussion on both #21201 (the original issue that triggered the change)
and #22746 (discussing whether to revert the change) failed to identify
a single motivating example for this change. To avoid breaking existing
users for no clear reason, revert the change.

The Reader has been rewritten in the interim so this is not a git revert
but instead and adjustment (and slight simplification) of the new Reader.

Fixes #22746.

Change-Id: Ie857b2f4b1359a207d085b6d3c3a6d440a997d12
Reviewed-on: https://go-review.googlesource.com/78295
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2017-11-16 16:29:37 +00:00
Russ Cox d0ce197c58 encoding/hex: make Decode, Decoder, DecodeString agree about partial results and errors
CL 70210 added Decoder for #21590, and in doing so it changed
the existing func Decode to return partial results for decoding errors.
That seems like a good change to make to Decode, but it was
untested (except as used by Decoder), inconsistent with DecodeString
in all error cases, and inconsistent with Decoder in not returning
partial results for odd-length input strings.

This CL makes Decode, DecodeString, and Decoder all agree about
the handling of partial results (they are returned) and error
precedence (the error earliest in the input is reported),
and it documents and tests this.

Change-Id: Ifb7d1e100ecb66fe2ed5ba34a621084d480f16db
Reviewed-on: https://go-review.googlesource.com/78120
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-11-16 01:08:20 +00:00
Hiroshi Ioka c32626a4ce encoding/asn1: add MarshalWithParams
Fixes #18873

Change-Id: Idb9750f739f91ebca34efcbc177254d412b4d90d
Reviewed-on: https://go-review.googlesource.com/44111
Reviewed-by: Adam Langley <agl@golang.org>
Run-TryBot: Adam Langley <agl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-11-16 00:37:47 +00:00
Russ Cox 894869e150 encoding/xml: move unexported const out of exported const block
CL 58210 introduced this constant for reasons I don't understand.
It should not be in the exported const block, which will pollute
godoc output with a "... unexported" notice.

Also since we already have a constant named xmlnsPrefix for "xmlns",
it is very confusing to also have xmlNamespacePrefix for "xml".
If we must have the constant at all, rename it to xmlPrefix.

Change-Id: I15f937454d730005816fcd32b1acca703acf1e51
Reviewed-on: https://go-review.googlesource.com/78121
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-15 21:27:07 +00:00
Russ Cox c4c3f2a1f2 encoding/csv: rename ParseError.RecordLine to .StartLine
A record can span multiple lines (the whole reason for the extra field),
so the important fact is that it's the _start_ of the record.
Make that clear in the name.

(This API was added during the Go 1.10 cycle so it can still be cleaned up.)

Change-Id: Id95b3ceb7cdfc4aa0ed5a053cb84da8945fa5496
Reviewed-on: https://go-review.googlesource.com/78119
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2017-11-15 21:26:52 +00:00
Russ Cox 9232a612fe encoding/binary: make new example a bit more idiomatic
Mainly get rid of the weird zero-value struct literal,
but while we're here also group and order things a bit better:
first the reader, then the data, then the call (which takes reader then data).

Change-Id: I901b0661d85d8eaa0807e4482aac66500ca996c7
Reviewed-on: https://go-review.googlesource.com/78118
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
2017-11-15 21:26:43 +00:00
Tim Cooper 707a4d3fed encoding/pem: add Encode example
Change-Id: Ib9ec3524b712e016a9dd2fbee5555362c1a0cb59
Reviewed-on: https://go-review.googlesource.com/77770
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-11-15 03:08:55 +00:00
Tim Cooper f4f6018d38 encoding/pem: add Encode, EncodeToMemory docs
Included in a warning that EncodeToMemory may return an incomplete PEM
encoded structure if invalid headers are supplied. Example:

	pem.EncodeToMemory(&pem.Block{
		Headers: map[string]string{
			"a":   "test1",
			"b:c": "test2",
		},
	})

Returns:

	-----BEGIN -----
	a: test1

Change-Id: Ia9cf0202f985e3cf210aabb6f07667e581ff081f
Reviewed-on: https://go-review.googlesource.com/77790
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-15 02:35:09 +00:00
Joe Tsai 0cee4b7b78 encoding/json: always ignore embedded pointers to unexported struct types
CL 60410 fixes a bug in reflect that allows assignments to an embedded
field of a pointer to an unexported struct type.
This breaks the json package because unmarshal is now unable to assign
a newly allocated struct to such fields.

In order to be consistent in the behavior for marshal and unmarshal,
this CL changes both marshal and unmarshal to always ignore
embedded pointers to unexported structs.

Fixes #21357

Change-Id: If62ea11155555e61115ebb9cfa5305caf101bde5
Reviewed-on: https://go-review.googlesource.com/76851
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-11-13 18:23:38 +00:00
Ian Lance Taylor ed3d672766 encoding/json: permit encoding uintptr as a string
Fixes #22629

Change-Id: I31e85f9faa125ee0dfd6d3c5fa89334b00d61e6e
Reviewed-on: https://go-review.googlesource.com/76530
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Joe Tsai <joetsai@google.com>
2017-11-08 20:34:55 +00:00
Leigh McCulloch 65a864a628 encoding/xml: add Marshal doc about name conflicts
The docs for xml.Marshal state that the XML elements name is derived
from one of five locations in a specific order of precedence, but does
not mention that if the field is a struct type and has its name defined
in a tag and in the types XMLName field that an error will occur. This
is documented in the structFieldInfo function but not in the function
documentation, and the existing docs in Marshal are misleading without
this behavior being discussed.

Fixes #18564

Change-Id: I29042f124a534bd1bc993f1baeddaa0af2e72fed
Reviewed-on: https://go-review.googlesource.com/76321
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-11-08 18:05:39 +00:00
Leigh McCulloch 8db19a4966 all: change github.com issue links to golang.org
The go repository contains a mix of github.com/golang/go/issues/xxxxx
and golang.org/issues/xxxxx URLs for references to issues in the issue
tracker. We should use one for consistency, and golang.org is preferred
in case the project moves the issue tracker in the future.

This reasoning is taken from a comment Sam Whited left on a CL I
recently opened: https://go-review.googlesource.com/c/go/+/73890.

In that CL I referenced an issue using its github.com URL, because other
tests in the file I was changing contained references to issues using
their github.com URL. Sam Whited left a comment on the CL stating I
should change it to the golang.org URL.

If new code is intended to reference issues via golang.org and not
github.com, existing code should be updated so that precedence exists
for contributors who are looking at the existing code as a guide for the
code they should write.

Change-Id: I3b9053fe38a1c56fc101a8b7fd7b8f310ba29724
Reviewed-on: https://go-review.googlesource.com/75673
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2017-11-04 04:13:41 +00:00
Lynn Boger bb1fd3b5ff cmd/compile: add rules to improve consecutive byte loads and stores on ppc64le
This adds new rules to recognize consecutive byte loads and
stores and lowers them to loads and stores such as lhz, lwz, ld,
sth, stw, std. This change only covers the little endian cases
on little endian machines, such as is found in encoding/binary
UintXX or PutUintXX for little endian. Big endian will be done
later.

Updates were also made to binary_test.go to allow the benchmark
for Uint and PutUint to actually use those functions because
the way they were written, those functions were being
optimized out.

Testcases were also added to cmd/compile/internal/gc/asm_test.go.

Updates #22496

The following improvement can be found in golang.org/x/crypto

poly1305:

Benchmark64-16              142           114           -19.72%
Benchmark1K-16              1717          1424          -17.06%
Benchmark64Unaligned-16     142           113           -20.42%
Benchmark1KUnaligned-16     1721          1428          -17.02%

chacha20poly1305:

BenchmarkChacha20Poly1305Open_64-16     1012       885   -12.55%
BenchmarkChacha20Poly1305Seal_64-16     971        836   -13.90%
BenchmarkChacha20Poly1305Open_1350-16   11113      9539  -14.16%
BenchmarkChacha20Poly1305Seal_1350-16   11013      9392  -14.72%
BenchmarkChacha20Poly1305Open_8K-16     61074      53431 -12.51%
BenchmarkChacha20Poly1305Seal_8K-16     61214      54806 -10.47%

Other improvements of around 10% found in crypto/tls.

Results after updating encoding/binary/binary_test.go:

BenchmarkLittleEndianPutUint64-16     1.87      0.93      -50.27%
BenchmarkLittleEndianPutUint32-16     1.19      0.93      -21.85%
BenchmarkLittleEndianPutUint16-16     1.16      1.03      -11.21%

Change-Id: I7bbe2fbcbd11362d58662fecd907a0c07e6ca2fb
Reviewed-on: https://go-review.googlesource.com/74410
Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Michael Munday <mike.munday@ibm.com>
2017-11-03 18:46:59 +00:00
Joe Tsai 08f19bbde1 go/printer: forbid empty line before first comment in block
To improve readability when exported fields are removed,
forbid the printer from emitting an empty line before the first comment
in a const, var, or type block.
Also, when printing the "Has filtered or unexported fields." message,
add an empty line before it to separate the message from the struct
or interfact contents.

Before the change:
<<<
type NamedArg struct {

        // Name is the name of the parameter placeholder.
        //
        // If empty, the ordinal position in the argument list will be
        // used.
        //
        // Name must omit any symbol prefix.
        Name string

        // Value is the value of the parameter.
        // It may be assigned the same value types as the query
        // arguments.
        Value interface{}
        // contains filtered or unexported fields
}
>>>

After the change:
<<<
type NamedArg struct {
        // Name is the name of the parameter placeholder.
        //
        // If empty, the ordinal position in the argument list will be
        // used.
        //
        // Name must omit any symbol prefix.
        Name string

        // Value is the value of the parameter.
        // It may be assigned the same value types as the query
        // arguments.
        Value interface{}

        // contains filtered or unexported fields
}
>>>

Fixes #18264

Change-Id: I9fe17ca39cf92fcdfea55064bd2eaa784ce48c88
Reviewed-on: https://go-review.googlesource.com/71990
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
2017-11-02 18:17:22 +00:00
Leigh McCulloch 14bc4f5e5f encoding/xml: ignore whitespace in values and attrs
Whitespace is ignored in bool values and attrs. It is convenient and
relatively safe since whitespace around a bool value is often
unimportant. The same logic can be applied to numeric values of types
int, uint, and float.

Fixes #22146

Change-Id: Ie0462def90304af144b8e2e72d85b644857c27cc
Reviewed-on: https://go-review.googlesource.com/73891
Reviewed-by: Sam Whited <sam@samwhited.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Sam Whited <sam@samwhited.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-11-01 20:39:57 +00:00
Leigh McCulloch 776cdefc07 encoding/xml: add docs and tests for bool whitespace
Whitespace is ignored in bool values and attrs, but there are no tests
capturing this behavior.

Change-Id: I7a7249de4886f510869e91de937e69b83c3254c8
Reviewed-on: https://go-review.googlesource.com/73890
Reviewed-by: Sam Whited <sam@samwhited.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Sam Whited <sam@samwhited.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-11-01 20:35:59 +00:00
Michael Fraenkel f1ce59d988 encoding/json: Include the offset of a SyntaxError
When a SyntaxError occurs, report the current offset within the stream.
The code already accounted for the offset within the current buffer
being scanned. By including how much data was already scanned, the
current offset can be computed.

Fixes #22478

Change-Id: I91ecd4cad0b85a5c1556bc597f3ee914e769af01
Reviewed-on: https://go-review.googlesource.com/74251
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-10-31 22:44:14 +00:00
Ivan Bertona 2596a0c075 encoding/json: disallow unknown fields in Decoder
Add a DisallowUnknownFields flag to Decoder.

DisallowUnknownFields causes the Decoder to return an error when
the the decoding destination is a struct and the input contains
object keys which do not match any non-ignored, public field the
destination, including keys whose value is set to null.

Note: this fix has already been worked on in 27231, which seems
to be abandoned. This version is a slightly simpler implementation
and is up to date with the master branch.

Fixes #15314

Change-Id: I987a5857c52018df334f4d1a2360649c44a7175d
Reviewed-on: https://go-review.googlesource.com/74830
Reviewed-by: Joe Tsai <joetsai@google.com>
Run-TryBot: Joe Tsai <joetsai@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-10-31 22:28:36 +00:00
Sam Whited be08ddbfcd encoding/xml: don't panic when custom Unmarshaler sees StartElement
Change-Id: I90aa0a983abd0080f3de75d3340fdb15c1f9ca35
Reviewed-on: https://go-review.googlesource.com/70891
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Sam Whited <sam@samwhited.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-10-30 18:52:19 +00:00
Joe Tsai e7fed7fa35 encoding/csv: forbid certain Comma and Comment runes
The record delimiter (not configurable by user) is "\r\n" or "\n".
It is insensible for the user to set Comma or Comment delimiters
to be some character that conflicts with the record delimiter.
Furthermore, it is insensible for Comma or Comment to be the same rune.
Allowing this leaks implementation details to the user in regards to
the evaluation order of which rune is checked for first.

Fixes #22404

Change-Id: I31e86abc9b3a8fb4584e090477795587740970ae
Reviewed-on: https://go-review.googlesource.com/72793
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-10-25 01:43:46 +00:00
Joe Tsai 8c532f5fc4 encoding/csv: update ErrQuote message
The ErrQuote variable is only returned when a parsing error
occurs within a quoted string. Make that clear in the message.

Change-Id: I06ad5a9edb41afedde193c4f8b93551bb8342bbb
Reviewed-on: https://go-review.googlesource.com/72794
Reviewed-by: Avelino <t@avelino.xxx>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-10-24 07:10:28 +00:00
Joe Tsai 744da64947 encoding/csv: fix error documentation
We should be referring to ParseError.Err, which is the underlying error,
not ParseError.Error, which is the error method.

Change-Id: Ic3cef5ecbe1ada5fa14b9573222f29da8fc9a8d5
Reviewed-on: https://go-review.googlesource.com/72450
Reviewed-by: Tim Cooper <tim.cooper@layeh.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-10-24 07:10:10 +00:00
Joe Tsai 29ea82d072 encoding/csv: add ParseError.RecordLine
CL 72150 fixes #22352 by reverting the problematic parts of that CL
where the line number and column number were inconsistent with each other.
This CL adds back functionality to address the issue that CL 72150
was trying to solve in the first place. That is, it reports the starting
line of the record, so that users have a frame of reference to start with
when debugging what went wrong.

In the event of gnarly CSV files with multiline quoted strings, a parse
failure likely occurs somewhere between the start of the record and
the point where the parser finally detected an error.
Since ParserError.{Line,Column} reports where the *error* occurs, we
add a RecordLine field to report where the record starts.

Also take this time to cleanup and modernize TestRead.

Fixes #19019
Fixes #22352

Change-Id: I16cebf0b81922c35f75804c7073e9cddbfd11a04
Reviewed-on: https://go-review.googlesource.com/72310
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-10-21 01:32:28 +00:00
Tim Cooper 6db4950dc5 encoding/hex: add NewEncoder, NewDecoder
NewEncoder returns an io.Writer that writes all incoming bytes as
hexadecimal characters to the underlying io.Writer. NewDecoder returns an
io.Reader that does the inverse.

Fixes #21590

Change-Id: Iebe0813faf365b42598f19a9aa41768f571dc0a8
Reviewed-on: https://go-review.googlesource.com/70210
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-10-20 23:47:07 +00:00
Joe Tsai 89ccfe4962 encoding/csv: simplify and optimize Reader
The Reader implementation is slow because it operates on a rune-by-rune
basis via bufio.Reader.ReadRune. We speed this up by operating on entire
lines that we read from bufio.Reader.ReadSlice.

In order to ensure that we read the full line, we augment ReadSlice
in our Reader.readLine method to automatically expand the slice if
bufio.ErrBufferFull is every hit.

This change happens to fix #19410 because it no longer relies on
rune-by-rune parsing and only searches for the relevant delimiter rune.

In order to keep column accounting simple and consistent, this change
reverts parts of CL 52830.

This CL is an alternative to CL 36270 and builds on some of the ideas
from that change by Diogo Pinela.

name                                     old time/op    new time/op    delta
Read-8                                   3.12µs ± 1%    2.54µs ± 2%  -18.76%   (p=0.000 n=10+9)
ReadWithFieldsPerRecord-8                3.12µs ± 1%    2.53µs ± 1%  -18.91%    (p=0.000 n=9+9)
ReadWithoutFieldsPerRecord-8             3.13µs ± 0%    2.57µs ± 3%  -18.07%  (p=0.000 n=10+10)
ReadLargeFields-8                        52.3µs ± 1%     5.3µs ± 2%  -89.93%   (p=0.000 n=10+9)
ReadReuseRecord-8                        2.05µs ± 1%    1.40µs ± 1%  -31.48%   (p=0.000 n=10+9)
ReadReuseRecordWithFieldsPerRecord-8     2.05µs ± 1%    1.41µs ± 0%  -31.03%   (p=0.000 n=10+9)
ReadReuseRecordWithoutFieldsPerRecord-8  2.06µs ± 1%    1.40µs ± 1%  -31.70%   (p=0.000 n=9+10)
ReadReuseRecordLargeFields-8             50.9µs ± 0%     4.1µs ± 3%  -92.01%  (p=0.000 n=10+10)

name                                     old alloc/op   new alloc/op
Read-8                                       664B ± 0%      664B ± 0%
ReadWithFieldsPerRecord-8                    664B ± 0%      664B ± 0%
ReadWithoutFieldsPerRecord-8                 664B ± 0%      664B ± 0%
ReadLargeFields-8                          3.94kB ± 0%    3.94kB ± 0%
ReadReuseRecord-8                           24.0B ± 0%     24.0B ± 0%
ReadReuseRecordWithFieldsPerRecord-8        24.0B ± 0%     24.0B ± 0%
ReadReuseRecordWithoutFieldsPerRecord-8     24.0B ± 0%     24.0B ± 0%
ReadReuseRecordLargeFields-8               2.98kB ± 0%    2.98kB ± 0%

name                                     old allocs/op  new allocs/op
Read-8                                       18.0 ± 0%      18.0 ± 0%
ReadWithFieldsPerRecord-8                    18.0 ± 0%      18.0 ± 0%
ReadWithoutFieldsPerRecord-8                 18.0 ± 0%      18.0 ± 0%
ReadLargeFields-8                            24.0 ± 0%      24.0 ± 0%
ReadReuseRecord-8                            8.00 ± 0%      8.00 ± 0%
ReadReuseRecordWithFieldsPerRecord-8         8.00 ± 0%      8.00 ± 0%
ReadReuseRecordWithoutFieldsPerRecord-8      8.00 ± 0%      8.00 ± 0%
ReadReuseRecordLargeFields-8                 12.0 ± 0%      12.0 ± 0%

Updates #22352
Updates #19019
Fixes #16791
Fixes #19410

Change-Id: I31c27cfcc56880e6abac262f36c947179b550bbf
Reviewed-on: https://go-review.googlesource.com/72150
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-10-20 23:20:46 +00:00
Joe Tsai a3e013b082 encoding/json: use Deprecated markers
In #10909, it was decided that "Deprecated:" is a magic string for
tools (e.g., #17056 for godoc) to detect deprecated identifiers.
Use those convention instead of custom written prose.

Change-Id: Ia514fc3c88fc502e86c6e3de361c435f4cb80b22
Reviewed-on: https://go-review.googlesource.com/70110
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
2017-10-11 22:09:22 +00:00
Adam Langley 3b186db7b4 encoding/asn1: don't encode strings with '*' as PrintableString.
The '*' character is not allowed in ASN.1 PrintableString. However, due
to wide-spread use, we permit it so that we can parse many certificates
with wildcards. However, that also meant that generic strings with
asterisks in would be encoded as PrintableString.

This change makes the default for such strings to be UTF8String. Thus,
while the code PrintableStrings containing '*', it will not generate
them unless the string type was specified in the struct field tag.

Change-Id: I2d458da36649427352eeaa50a1b6020108b2ccbd
Reviewed-on: https://go-review.googlesource.com/68990
Reviewed-by: Adam Langley <agl@golang.org>
2017-10-09 19:28:47 +00:00
Josselin Costanzi 244c98fe01 encoding/base64: optimize DecodeString
Optimize base64 decoding speed by adding 32-bits and 64-bits specialized
methods that don't perform any error checking and fall back to the more
complex decodeQuantum method when a non-base64 character is present.

On a 64-bits cpu:

name                 old time/op    new time/op     delta
DecodeString/2-4       70.0ns ± 6%     69.2ns ± 0%     ~     (p=0.169 n=5+8)
DecodeString/4-4       91.3ns ± 2%     80.4ns ± 0%  -11.89%  (p=0.001 n=5+10)
DecodeString/8-4        126ns ± 5%      106ns ± 0%  -16.14%  (p=0.000 n=5+7)
DecodeString/64-4       652ns ±21%      361ns ± 0%  -44.57%  (p=0.000 n=5+7)
DecodeString/8192-4    61.0µs ±13%     31.5µs ± 1%  -48.38%  (p=0.001 n=5+9)

name                 old speed      new speed       delta
DecodeString/2-4     57.2MB/s ± 6%   57.7MB/s ± 2%     ~     (p=0.419 n=5+9)
DecodeString/4-4     87.7MB/s ± 2%   99.5MB/s ± 0%  +13.45%  (p=0.001 n=5+10)
DecodeString/8-4     94.8MB/s ± 5%  112.6MB/s ± 1%  +18.82%  (p=0.001 n=5+9)
DecodeString/64-4     136MB/s ±19%    243MB/s ± 0%  +78.17%  (p=0.003 n=5+7)
DecodeString/8192-4   180MB/s ±11%    347MB/s ± 1%  +92.94%  (p=0.001 n=5+9)

Improves #19636

Change-Id: Ic10a454851093a7e1d46ca0c140deed73535d990
Reviewed-on: https://go-review.googlesource.com/38632
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-10-09 15:39:51 +00:00
Laurent Voisin dd4988fd99 encoding/gob: fix a typo in the package documentation
Change-Id: I6aee75a6ae1470dfc658d72a5b6508fb93503057
Reviewed-on: https://go-review.googlesource.com/69115
Reviewed-by: Alberto Donizetti <alb.donizetti@gmail.com>
2017-10-08 18:11:32 +00:00