A Directive (like <!ENTITY xxx []>) can't have other nodes nested inside
it (in our data structure representation), so there is no way to
preserve comments. The previous behavior was to just elide them, which
however might change the semantic meaning of the surrounding markup.
Instead, replace them with a space which hopefully has the same semantic
effect of the comment.
Directives are not actually a node type in the XML spec, which instead
specifies each of them separately (<!ENTITY, <!DOCTYPE, etc.), each with
its own grammar. The rules for where and when the comments are allowed
are not straightforward, and can't be implemented without implementing
custom logic for each of the directives.
Simply preserving the comments in the body of the directive would be
problematic, as there can be unmatched quotes inside the comment.
Whether those quotes are considered meaningful semantically or not,
other parsers might disagree and interpret the output differently.
This issue was reported by Juho Nurminen of Mattermost as it leads to
round-trip mismatches. See #43168. It's not being fixed in a security
release because round-trip stability is not a currently supported
security property of encoding/xml, and we don't believe these fixes
would be sufficient to reliably guarantee it in the future.
Fixes CVE-2020-29510
Updates #43168
Change-Id: Icd86c75beff3e1e0689543efebdad10ed5178ce3
Reviewed-on: https://go-review.googlesource.com/c/go/+/277893
Run-TryBot: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Filippo Valsorda <filippo@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
Before this change, <:name> would parse as <name>, which could cause
issues in applications that rely on the parse-encode cycle to
round-trip. Similarly, <x name:=""> would parse as expected but then
have the attribute dropped when serializing because its name was empty.
Finally, <a🅱️c> would parse and get serialized incorrectly. All these
values are invalid XML, but to minimize the impact of this change, we
parse them whole into Name.Local.
This issue was reported by Juho Nurminen of Mattermost as it leads to
round-trip mismatches. See #43168. It's not being fixed in a security
release because round-trip stability is not a currently supported
security property of encoding/xml, and we don't believe these fixes
would be sufficient to reliably guarantee it in the future.
Fixes CVE-2020-29509
Fixes CVE-2020-29511
Updates #43168
Change-Id: I68321c4d867305046f664347192948a889af3c7f
Reviewed-on: https://go-review.googlesource.com/c/go/+/277892
Run-TryBot: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Filippo Valsorda <filippo@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
This change properly handles a TokenReader which
returns an EOF in the middle of an open XML
element.
Thanks to Sam Whited for reporting this.
Fixes CVE-2021-27918
Fixes#44913
Change-Id: Id02a3f3def4a1b415fa2d9a8e3b373eb6cb0f433
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/1004594
Reviewed-by: Russ Cox <rsc@google.com>
Reviewed-by: Roland Shoemaker <bracewell@google.com>
Reviewed-by: Filippo Valsorda <valsorda@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/300391
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
The IsExported method is a more intuitive helper for checking whether
the method or field is exported than checking whether PkgPath is empty.
In the same CL, modify the standard library to make use of this helper.
Fixes#41563
Change-Id: Iaacfb3b74449501f98e2707aa32095a32bd3c3c1
Reviewed-on: https://go-review.googlesource.com/c/go/+/266197
Trust: Joe Tsai <joetsai@digital-static.net>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Fixes the check for the reserved namespace prefix
"xml" to be case insensitive, so as to match all variants of:
(('X'|'x')('M'|'m')('L'|'l'))
as mandated by Section 2.3 of https://www.w3.org/TR/REC-xml/
This is a roll forward of CL 203417, which was rolled back by CL 240179.
We've decided that the roll back was incorrect, and any broken tests
should be fixed.
The original CL 203417 was by Tamás Gulácsi.
Fixes#35151
For #39876
Change-Id: I2e6daa7aeb252531fba0b8a56086613e13059528
Reviewed-on: https://go-review.googlesource.com/c/go/+/264024
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
This reverts CL 203417.
Reason for revert: This change changes uses of tags like "XMLSchema-instance" without any recourse.
For #35151Fixes#39876
Change-Id: I4c85c8267a46b3748664b5078794dafffb42aa26
Reviewed-on: https://go-review.googlesource.com/c/go/+/240179
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
fieldInfo.value used to initialize nil anonymous struct fields if they
were encountered. This behavior is wanted when decoding, but not when
encoding. When encoding, the value should never be modified, and these
nil fields should be skipped entirely.
To fix the bug, add a bool argument to the function which tells the
code whether we are encoding or decoding.
Finally, add a couple of tests to cover the edge cases pointed out in
the original issue.
Fixes#27240.
Change-Id: Ic97ae4bfe5f2062c8518e03d1dec07c3875e18f6
Reviewed-on: https://go-review.googlesource.com/c/go/+/196809
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Fixes the check for the reserved namespace prefix
"xml" to be case insensitive, so as to match all variants of:
(('X'|'x')('M'|'m')('L'|'l'))
as mandated by Section 2.3 of https://www.w3.org/TR/REC-xml/Fixes#35151.
Change-Id: Id5a98e5f9d69d3741dc16f567c4320f1ad0b3c70
Reviewed-on: https://go-review.googlesource.com/c/go/+/203417
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Instead use string(r) where r has type rune.
This is in preparation for a vet warning for string(i).
Updates #32479
Change-Id: Ic205269bba1bd41723950219ecfb67ce17a7aa79
Reviewed-on: https://go-review.googlesource.com/c/go/+/220844
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Akhil Indurti <aindurti@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Toshihiro Shiino <shiino.toshihiro@gmail.com>
The documentation for TokenReader suggests that implementations of the
interface may return a token and io.EOF together, indicating that it is
the last token in the stream. This is similar to io.Reader. However, if
you wrap such a TokenReader in a Decoder it complained about the EOF.
A test was added to ensure this behavior on Decoder's.
Change-Id: I9083c91d9626180d3bcf5c069a017050f3c7c4a8
Reviewed-on: https://go-review.googlesource.com/c/go/+/130556
Run-TryBot: Sam Whited <sam@samwhited.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Some were never used, and some haven't been used for years.
One exception is net/http's readerAndCloser, which was only used in a
test. Move it to a test file.
While at it, remove a check in regexp that could never fire; the field
is an uint32, so it can never be negative.
Change-Id: Ia2200f6afa106bae4034045ea8233b452f38747b
Reviewed-on: https://go-review.googlesource.com/c/go/+/192621
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Per the code review guidelines: "Words in names that are
initialisms or acronyms have a consistent case."
Change-Id: I347b02d2f48455f2cbbc040191ba197e3e8f23fc
Reviewed-on: https://go-review.googlesource.com/c/go/+/191970
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Both the encoding/xml and encoding/json packages support custom
marshalers for JSON and XML, as well as the basic encoding.TextMarshaler
and encoding.TextUnmarshaler interfaces, but the docs and examples for
these are missing.
There are docs for how to use encoding.TextMarshaler and
encoding.TextUnmarshaler in encoding/json, but not encoding/xml. There
are no examples for how to use them with either json or xml. This commit
includes docs for encoding/xml and examples for both encoding/json and
encoding/xml.
There is an example using custom marshalers MarshalJSON and
UnmarshalJSON in encoding/json, but not MarshalXML and UnmarshalXML in
encoding/json. These docs are more so necessary for encoding/xml because
the complexities of XML documents is significantly greater than JSON
documents which more often leads to the need for custom marshaling. The
encoding/json package includes an example of how to write a custom
marshaler, and this commit includes the same example for the xml
package.
All examples are mirrored off the existing custom marshaler example in
encoding/json.
Fixes#6859
Change-Id: Ic93abc27c0b4d5e48dea6ede4e20b1bedca4ab39
Reviewed-on: https://go-review.googlesource.com/c/76350
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
They didn't even have public types, which made them pretty mysterious.
Give them types and reference the Decoder, which uses them.
Also, refer them qualified by their package name in the examples, as
we usually do in example*.go files, which usually use package foo_test
specifically so we can show the package names along with the symbols.
Change-Id: I50ebbbf43778c1627bfa526f8824f52c7953454f
Reviewed-on: https://go-review.googlesource.com/127663
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Each URL was manually verified to ensure it did not serve up incorrect
content.
Change-Id: I4dc846227af95a73ee9a3074d0c379ff0fa955df
Reviewed-on: https://go-review.googlesource.com/115798
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Section 2.2 of the referenced spec http://www.xml.com/axml/testaxml.htm
defines 0xD7FF as a (sub)range boundary, not 0xDF77.
Fixes#25172
Change-Id: Ic5a3328cd46ef6474b8e93c4a343dcfba0e6511f
Reviewed-on: https://go-review.googlesource.com/109495
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
It appears that old code (from 2009) in xml.(*Decoder).rawToken
replicates append's slice-growing functionality by allocating a new,
bigger backing array and then calling copy.
Simplifying the code by replacing it with a single append call does
not seem to hurt performance:
name old time/op new time/op delta
Marshal-4 11.2µs ± 1% 11.3µs ±10% ~ (p=0.069 n=19+17)
Unmarshal-4 28.6µs ± 1% 28.4µs ± 1% -0.60% (p=0.000 n=20+18)
name old alloc/op new alloc/op delta
Marshal-4 5.78kB ± 0% 5.78kB ± 0% ~ (all equal)
Unmarshal-4 8.61kB ± 0% 8.27kB ± 0% -3.90% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
Marshal-4 23.0 ± 0% 23.0 ± 0% ~ (all equal)
Unmarshal-4 189 ± 0% 190 ± 0% +0.53% (p=0.000 n=20+20)
Change-Id: Ie580d1216a44760e611e63dee2c339af5465aea5
Reviewed-on: https://go-review.googlesource.com/86655
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
CL 58210 introduced this constant for reasons I don't understand.
It should not be in the exported const block, which will pollute
godoc output with a "... unexported" notice.
Also since we already have a constant named xmlnsPrefix for "xmlns",
it is very confusing to also have xmlNamespacePrefix for "xml".
If we must have the constant at all, rename it to xmlPrefix.
Change-Id: I15f937454d730005816fcd32b1acca703acf1e51
Reviewed-on: https://go-review.googlesource.com/78121
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The docs for xml.Marshal state that the XML elements name is derived
from one of five locations in a specific order of precedence, but does
not mention that if the field is a struct type and has its name defined
in a tag and in the types XMLName field that an error will occur. This
is documented in the structFieldInfo function but not in the function
documentation, and the existing docs in Marshal are misleading without
this behavior being discussed.
Fixes#18564
Change-Id: I29042f124a534bd1bc993f1baeddaa0af2e72fed
Reviewed-on: https://go-review.googlesource.com/76321
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The go repository contains a mix of github.com/golang/go/issues/xxxxx
and golang.org/issues/xxxxx URLs for references to issues in the issue
tracker. We should use one for consistency, and golang.org is preferred
in case the project moves the issue tracker in the future.
This reasoning is taken from a comment Sam Whited left on a CL I
recently opened: https://go-review.googlesource.com/c/go/+/73890.
In that CL I referenced an issue using its github.com URL, because other
tests in the file I was changing contained references to issues using
their github.com URL. Sam Whited left a comment on the CL stating I
should change it to the golang.org URL.
If new code is intended to reference issues via golang.org and not
github.com, existing code should be updated so that precedence exists
for contributors who are looking at the existing code as a guide for the
code they should write.
Change-Id: I3b9053fe38a1c56fc101a8b7fd7b8f310ba29724
Reviewed-on: https://go-review.googlesource.com/75673
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Whitespace is ignored in bool values and attrs. It is convenient and
relatively safe since whitespace around a bool value is often
unimportant. The same logic can be applied to numeric values of types
int, uint, and float.
Fixes#22146
Change-Id: Ie0462def90304af144b8e2e72d85b644857c27cc
Reviewed-on: https://go-review.googlesource.com/73891
Reviewed-by: Sam Whited <sam@samwhited.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Sam Whited <sam@samwhited.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Whitespace is ignored in bool values and attrs, but there are no tests
capturing this behavior.
Change-Id: I7a7249de4886f510869e91de937e69b83c3254c8
Reviewed-on: https://go-review.googlesource.com/73890
Reviewed-by: Sam Whited <sam@samwhited.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Sam Whited <sam@samwhited.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
strings.LastIndexByte was introduced in go1.5 and it can be used
effectively wherever the second argument to strings.LastIndex is
exactly one byte long.
This avoids generating unnecessary string symbols and saves
a few calls to strings.LastIndex.
Change-Id: I7b5679d616197b055cffe6882a8675d24a98b574
Reviewed-on: https://go-review.googlesource.com/66372
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
strings.IndexByte was introduced in go1.2 and it can be used
effectively wherever the second argument to strings.Index is
exactly one byte long.
This avoids generating unnecessary string symbols and saves
a few calls to strings.Index.
Change-Id: I1ab5edb7c4ee9058084cfa57cbcc267c2597e793
Reviewed-on: https://go-review.googlesource.com/65930
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Went mainly for the ones that make no sense, such as the ones
mid-sentence or after commas.
Change-Id: Ie245d2c19cc7428a06295635cf6a9482ade25ff0
Reviewed-on: https://go-review.googlesource.com/57293
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
When unmarshaling, if an element is empty, eg. '<tag></tag>', and
destination type is int, uint, float or bool, do not attempt to parse
value (""). Set to its zero value instead.
Fixes#13417
Change-Id: I2d79f6d8f39192bb277b1a9129727d5abbb2dd1f
Reviewed-on: https://go-review.googlesource.com/38386
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Change-Id: I2d155c838935cd8427abd142a462ff4c56829715
Reviewed-on: https://go-review.googlesource.com/37948
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Found by github.com/mvdan/unparam.
Change-Id: I5a6664cceeba1cf1c2f3236ddf4db5ce7a64b02a
Reviewed-on: https://go-review.googlesource.com/37835
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The new tests in this CL have been checked against Go 1.7 as well
and all pass in Go 1.7, with the one exception noted in a comment
(an intentional change to omitempty already present before this CL).
CL 15684 made the intentional change to omitempty.
This CL fixes bugs introduced along the way.
Most of these are corner cases that are arguably not that important,
but they've always worked all the way back to Go 1, and someone
cared enough to file #19063. The most significant problem found
while adding tests is that in the case of a nil *string field with
`xml:",chardata"`, the existing code silently stops processing not just
that field but the entire remainder of the struct.
Even if #19063 were not worth fixing, this chardata bug would be.
Fixes#19063.
Change-Id: I318cf8f9945e1a4615982d9904e109fde577ebf9
Reviewed-on: https://go-review.googlesource.com/36954
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The tree is inconsistent about single l vs double l in those
words in documentation, test messages, and one error value text.
$ git grep -E '[Mm]arshall(|s|er|ers|ed|ing)' | wc -l
42
$ git grep -E '[Mm]arshal(|s|er|ers|ed|ing)' | wc -l
1694
Make it consistently a single l, per earlier decisions. This means
contributors won't be confused by misleading precedence, and it helps
consistency.
Change the spelling in one error value text in newRawAttributes of
crypto/x509 package to be consistent.
This change was generated with:
perl -i -npe 's,([Mm]arshal)l(|s|er|ers|ed|ing),$1$2,' $(git grep -l -E '[Mm]arshall' | grep -v AUTHORS | grep -v CONTRIBUTORS)
Updates #12431.
Follows https://golang.org/cl/14150.
Change-Id: I85d28a2d7692862ccb02d6a09f5d18538b6049a2
Reviewed-on: https://go-review.googlesource.com/33017
Run-TryBot: Minux Ma <minux@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
We only support unmarshaling into a string or a []byte, but we
previously would try (and panic while) setting a slice of a different
type. The docs say ",innerxml" is ignored if the type is not string or
[]byte, so do that for other slices as well.
Fixes#15600.
Change-Id: Ia64815945a14c3d04a0a45ccf413e38b58a69416
Reviewed-on: https://go-review.googlesource.com/32919
Run-TryBot: Quentin Smith <quentin@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
- Like ",any" for elements, add ",any,attr" for attributes to allow
a mop-up field that gets any otherwise unmapped attributes.
- Map attributes to fields of type slice by extending the slice,
just like for elements.
- Allow storing an attribute into an xml.Attr directly, to provide
a way to record the name.
Combined, these three independent features allow
AllAttrs []Attr `xml:",any,attr"`
to collect all attributes not otherwise spoken for in a particular struct.
Tests based on CL 16292 by Charles Weill.
Fixes#3633.
Change-Id: I2d75817f17ca8752d7df188080a407836af92611
Reviewed-on: https://go-review.googlesource.com/30946
Reviewed-by: Quentin Smith <quentin@golang.org>
No functional changes here. Just makes next CL easier to read.
Change-Id: Icf7b2281b4da6cb59ff4edff05943b2ee288576a
Reviewed-on: https://go-review.googlesource.com/30945
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
There was an inconsistency between the (json encoding + documentation)
and the xml encoding implementation. Pointer to an empty value was
not being serialized (i.e simply ignored). Which had the effect of making
impossible to have a struct with a string field for which we wanted to
serialize the value ""
Fixes#5452
Change-Id: Id858701801158409be01e962d2cda843424bd22a
Reviewed-on: https://go-review.googlesource.com/15684
Reviewed-by: Russ Cox <rsc@golang.org>
The error return from copyValue was ignored causing some XML attribute
parsing to swallow an error.
Additionally, type MyMarshalerAttrTest had no UnmarshalXMLAttr method
causing marshalTests not to be symmetrical and the test suite to fail
for test case 101.
Fixes#16158
Change-Id: Icebc505295a2c656ca4b42ba37bb0957dd7260c6
Reviewed-on: https://go-review.googlesource.com/27455
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Fixes#8833
Change-Id: I4523a1de112ed02371504e27882659bce8028a45
Reviewed-on: https://go-review.googlesource.com/24745
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Filippo Valsorda
John Bampton
Katie Hockman
Joe Tsai
Ian Lance Taylor
Daniel Martí
Tamás Gulácsi
Sam Whited
Eddie Scholtz
Leigh McCulloch
Brad Fitzpatrick
Tim Cooper
Artyom Pervukhin
tengufromsky
Ryuma Yoshida
Alberto Donizetti
Igor Vashyst
Russ Cox
Marvin Stenger
Karel Pazdera
Bryan C. Mills
Sarah Adams
Dmitri Shuralyov
Quentin Smith
Allan Simon
Eric Lagergren
Dominik Honnef