TestEnvOverride sets PATH to /wibble before executing a CGI.
So customized Perl that is starting with '#!/usr/bin/env bash' will fail
because /usr/bin/env can't lookup bash.
Fixes#27790
Change-Id: I25e433061a7ff9da8c86429e934418fc15f12f90
Reviewed-on: https://go-review.googlesource.com/c/go/+/196845
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Removes TestTimeoutHandlerAndFlusher due to flakes on
one of the builders due to timing issues.
Perhaps later, we might need to bring it back when we've
figured out the timing issues.
Fixes#34573.
Change-Id: Ia88d4da31fb228296144dc31f9a4288167fb4a53
Reviewed-on: https://go-review.googlesource.com/c/go/+/197757
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Also added a test to ensure that any interactions
between TimeoutHandler and Flusher result in the
correct status code and body, but also that we don't
get superfluous logs from stray writes as was seen
in the bug report.
Fixes#34439.
Change-Id: I4af62db256742326f9353f98a2fcb5f71d2a5fd9
Reviewed-on: https://go-review.googlesource.com/c/go/+/197659
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
RFC 7230 is clear about headers with a space before the colon, like
X-Answer : 42
being invalid, but we've been accepting and normalizing them for compatibility
purposes since CL 5690059 in 2012.
On the client side, this is harmless and indeed most browsers behave the same
to this day. On the server side, this becomes a security issue when the
behavior doesn't match that of a reverse proxy sitting in front of the server.
For example, if a WAF accepts them without normalizing them, it might be
possible to bypass its filters, because the Go server would interpret the
header differently. Worse, if the reverse proxy coalesces requests onto a
single HTTP/1.1 connection to a Go server, the understanding of the request
boundaries can get out of sync between them, allowing an attacker to tack an
arbitrary method and path onto a request by other clients, including
authentication headers unknown to the attacker.
This was recently presented at multiple security conferences:
https://portswigger.net/blog/http-desync-attacks-request-smuggling-reborn
net/http servers already reject header keys with invalid characters.
Simply stop normalizing extra spaces in net/textproto, let it return them
unchanged like it does for other invalid headers, and let net/http enforce
RFC 7230, which is HTTP specific. This loses us normalization on the client
side, but there's no right answer on the client side anyway, and hiding the
issue sounds worse than letting the application decide.
Fixes CVE-2019-16276
Fixes#34540
Change-Id: I6d272de827e0870da85d93df770d6a0e161bbcf1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/549719
Reviewed-by: Brad Fitzpatrick <bradfitz@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/197503
Run-TryBot: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
When the http2 transport returns a NoCachedConnError, the connection
must be removed from the idle list as well as the connections per host.
Fixes#34387
Change-Id: I7875c9c95e694a37a339bb04385243b49f9b20d3
Reviewed-on: https://go-review.googlesource.com/c/go/+/196665
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
CL 140357 caused HTTP/2 connections to be put in the idle pool, but
failed to properly guard the trace.GotConn call in getConn. dialConn
returns a minimal persistConn with conn == nil for HTTP/2 connections.
This persistConn was then returned from queueForIdleConn and caused the
httptrace.GotConnInfo passed into GotConn to have a nil Conn field.
HTTP/2 connections call GotConn themselves so leave it for HTTP/2 to call
GotConn as is done directly below.
Fixes#34282
Change-Id: If54bfaf6edb14f5391463f908efbef5bb8a5d78e
GitHub-Last-Rev: 2b7d66a1ce
GitHub-Pull-Request: golang/go#34283
Reviewed-on: https://go-review.googlesource.com/c/go/+/195237
Reviewed-by: Michael Fraenkel <michael.fraenkel@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Use the following (suboptimal) script to obtain a list of possible
typos:
#!/usr/bin/env sh
set -x
git ls-files |\
grep -e '\.\(c\|cc\|go\)$' |\
xargs -n 1\
awk\
'/\/\// { gsub(/.*\/\//, ""); print; } /\/\*/, /\*\// { gsub(/.*\/\*/, ""); gsub(/\*\/.*/, ""); }' |\
hunspell -d en_US -l |\
grep '^[[:upper:]]\{0,1\}[[:lower:]]\{1,\}$' |\
grep -v -e '^.\{1,4\}$' -e '^.\{16,\}$' |\
sort -f |\
uniq -c |\
awk '$1 == 1 { print $2; }'
Then, go through the results manually and fix the most obvious typos in
the non-vendored code.
Change-Id: I3cb5830a176850e1a0584b8a40b47bde7b260eae
Reviewed-on: https://go-review.googlesource.com/c/go/+/193848
Reviewed-by: Robert Griesemer <gri@golang.org>
Some were never used, and some haven't been used for years.
One exception is net/http's readerAndCloser, which was only used in a
test. Move it to a test file.
While at it, remove a check in regexp that could never fire; the field
is an uint32, so it can never be negative.
Change-Id: Ia2200f6afa106bae4034045ea8233b452f38747b
Reviewed-on: https://go-review.googlesource.com/c/go/+/192621
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The URL port is already checked in net/url, so we can remove the redundant validation in net/http.
Fixes#33600
Change-Id: I62511a452df6262d4b66180933758d34627ff9df
GitHub-Last-Rev: c19afa3102
GitHub-Pull-Request: golang/go#33611
Reviewed-on: https://go-review.googlesource.com/c/go/+/190057
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The net/url error in question now quotes the URL, so update the expected
output string.
While at it, also update a comment in httputil, though that doesn't
affect any test.
Fixes#33910.
Change-Id: I0981f528b24337c2952ef60c0db3b7ff72d72110
Reviewed-on: https://go-review.googlesource.com/c/go/+/192078
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Current implementation doesn't always make it obvious what the exact
problem with the URL is, so this makes it clearer by consistently quoting
the invalid URL, as is the norm in other parsing implementations, eg.:
strconv.Atoi(" 123") returns an error: parsing " 123": invalid syntax
Updates #29261
Change-Id: Icc6bff8b4a4584677c0f769992823e6e1e0d397d
GitHub-Last-Rev: 648b9d93fe
GitHub-Pull-Request: golang/go#29384
Reviewed-on: https://go-review.googlesource.com/c/go/+/185117
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
When an invalid URL was passed to DumpRequestOut, it would directly return
without gracefully shutting down the reader goroutine.
So we create a channel and signal the reader goroutine to exit
if an error occurs during roundtrip.
Fixes#32571
Change-Id: I8c2970f1601e599f3d1ebfed298faf5f5716fc2c
Reviewed-on: https://go-review.googlesource.com/c/go/+/182037
Run-TryBot: Agniva De Sarker <agniva.quicksilver@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
I'm trying to keep the code changes minimal for backporting to Go 1.13,
so it is still possible for a handful of entries to leak,
but the leaks are now O(1) instead of O(N) in the steady state.
Longer-term, I think it would be a good idea to coalesce idleMu with
connsPerHostMu and clear entries out of both queues as soon as their
goroutines are done waiting.
Fixes#33849Fixes#33850
Change-Id: Ia66bc64671eb1014369f2d3a01debfc023b44281
Reviewed-on: https://go-review.googlesource.com/c/go/+/191964
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
HTTP is an initialism, not an acronym, where you pronounce each letter as a
word. It's "an H", not "a H".
Running `find src/net/http -type f | xargs grep -n 'an HTTP' | wc -l` shows
that the "an HTTP" form is used 67 times across the `net/http` package.
Furthermore, `find src/net/http -type f | xargs grep -n 'a HTTP' | wc -l`
yields only 4 results.
Change-Id: I219c292a9e2c9bf7a009dbfe82ea8b15874685e9
GitHub-Last-Rev: 6ebd095023
GitHub-Pull-Request: golang/go#33810
Reviewed-on: https://go-review.googlesource.com/c/go/+/191700
Reviewed-by: Toshihiro Shiino <shiino.toshihiro@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The doc comments of both ServerContextKey and LocalAddrContextKey both suggest that context.WithValue can be used to access (get) properties of the server or connection. This PR fixes those comments to refer to Context.Value instead.
Change-Id: I4ed383ef97ba1951f90c555243007469cfc18d4d
GitHub-Last-Rev: 05bc3acf82
GitHub-Pull-Request: golang/go#33833
Reviewed-on: https://go-review.googlesource.com/c/go/+/191838
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Update golang.org/x/net to v0.0.0-20190813141303-74dc4d7220e7 to import
the following security fix.
commit 74dc4d7220e7acc4e100824340f3e66577424772
Author: Filippo Valsorda <filippo@golang.org>
Date: Sun Aug 11 02:12:18 2019 -0400
http2: limit number of control frames in server send queue
An attacker could cause servers to queue an unlimited number of PING
ACKs or RST_STREAM frames by soliciting them and not reading them, until
the program runs out of memory.
Limit control frames in the queue to a few thousands (matching the limit
imposed by other vendors) by counting as they enter and exit the scheduler,
so the protection will work with any WriteScheduler.
Once the limit is exceeded, close the connection, as we have no way to
communicate with the peer.
Change-Id: I842968fc6ed3eac654b497ade8cea86f7267886b
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/525552
Reviewed-by: Brad Fitzpatrick <bradfitz@google.com>
This change was generated with cmd/go and cmd/bundle:
$ go get -u golang.org/x/net
$ go mod tidy
$ go mod vendor
$ go generate net/http
Fixes CVE-2019-9512 and CVE-2019-9514
Fixes#33606
Change-Id: I464baf96175006aa101d65d3b0f6494f28a626ab
Reviewed-on: https://go-review.googlesource.com/c/go/+/190137
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
As of Go 1.13rc1, TimeoutHandler supports the Flusher and Pusher interfaces and
this change corrects its documentation to say that.
Fixes#33769
Updates #29193
Change-Id: Ia0523f7f2e3dc1f8f0b68950b85a7bf81c4abe60
GitHub-Last-Rev: 5310d2c960
GitHub-Pull-Request: golang/go#33770
Reviewed-on: https://go-review.googlesource.com/c/go/+/191237
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
When Host is not valid per RFC 3986, the behavior of Hostname and Port
was wildly unpredictable, to the point that Host could have a suffix
that didn't appear in neither Hostname nor Port.
This is a security issue when applications are applying checks to Host
and expecting them to be meaningful for the contents of Hostname.
To reduce disruption, this change only aims to guarantee the following
two security-relevant invariants.
* Host is either Hostname or [Hostname] with Port empty, or
Hostname:Port or [Hostname]:Port.
* Port is only decimals.
The second invariant is the one that's most likely to cause disruption,
but I believe it's important, as it's conceivable an application might
do a suffix check on Host and expect it to be meaningful for the
contents of Hostname (if the suffix is not a valid port).
There are three ways to ensure it.
1) Reject invalid ports in Parse. Note that non-numeric ports are
already rejected if and only if the host starts with "[".
2) Consider non-numeric ports as part of Hostname, not Port.
3) Allow non-numeric ports, and hope they only flow down to net/http,
which will reject them (#14353).
This change adopts both 1 and 2. We could do only the latter, but then
these invalid hosts would flow past port checks, like in
http_test.TestTransportRejectsAlphaPort. Non-numeric ports weren't fully
supported anyway, because they were rejected after IPv6 literals, so
this restores consistency. We could do only the former, but at this
point 2) is free and might help with manually constructed Host values
(or if we get something wrong in Parse).
Note that net.SplitHostPort and net.Dial explicitly accept service names
in place of port numbers, but this is an URL package, and RFC 3986,
Section 3.2.3, clearly specifies ports as a number in decimal.
net/http uses a mix of net.SplitHostPort and url.Parse that would
deserve looking into, but in general it seems that it will still accept
service names in Addr fields as they are passed to net.Listen, while
rejecting them in URLs, which feels correct.
This leaves a number of invalid URLs to reject, which however are not
security relevant once the two invariants above hold, so can be done in
Go 1.14: IPv6 literals without brackets (#31024), invalid IPv6 literals,
hostnames with invalid characters, and more.
Tested with 200M executions of go-fuzz and the following Fuzz function.
u, err := url.Parse(string(data))
if err != nil {
return 0
}
h := u.Hostname()
p := u.Port()
switch u.Host {
case h + ":" + p:
return 1
case "[" + h + "]:" + p:
return 1
case h:
fallthrough
case "[" + h + "]":
if p != "" {
panic("unexpected Port()")
}
return 1
}
panic("Host is not a variant of [Hostname]:Port")
Fixes CVE-2019-14809
Updates #29098
Change-Id: I7ef40823dab28f29511329fa2d5a7fb10c3ec895
Reviewed-on: https://go-review.googlesource.com/c/go/+/189258
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The bundle included changes from a commit after the one referred to by
the go.mod, probably due to cmd/bundle using the GOPATH source.
Identified with the new go/packages based cmd/bundle from CL 189818.
$ go get golang.org/x/net@461777fb6f
$ go mod tidy
$ go mod vendor
$ go generate net/http # with CL 189818
Also, updated the socks_bundle.go generate command to drop obsolete
options and match h2_bundle.go. It caused no output changes.
Updates #32031
Change-Id: I0322d4e842dbfdad749455111072ca4872a62ad4
Reviewed-on: https://go-review.googlesource.com/c/go/+/189897
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
It is unclear whether the current definition of os.IsTimeout is
desirable or not. Drop ErrTimeout for now so we can consider adding it
(or some other error) in a future release with a corrected definition.
Fixes#33411
Change-Id: I8b880da7d22afc343a08339eb5f0efd1075ecafe
Reviewed-on: https://go-review.googlesource.com/c/go/+/188758
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
As discussed in
https://github.com/golang/go/issues/32463#issuecomment-506833421
the classification of deadline-based timeouts as "temporary" errors is a
historical accident. I/O timeouts used to be duration-based, so they
really were temporary--retrying a timed-out operation could succeed. Now
that they're deadline-based, timeouts aren't temporary unless you reset
the deadline.
Drop ErrTemporary from Go 1.13, since its definition is wrong. We'll
consider putting it back in Go 1.14 with a clear definition and
deprecate net.OpError.Temporary.
Fixes#32463
Change-Id: I70cda664590d8872541e17409a5780da76920891
Reviewed-on: https://go-review.googlesource.com/c/go/+/188398
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Request.PostForm gets populated with form data for PATCH, POST, or PUT
http verbs.
Change-Id: I33065aa78a8470c4e9490aac830aa6f5963c61cb
Reviewed-on: https://go-review.googlesource.com/c/go/+/187821
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
In Go1.12 and below, the logic in ReverseProxy.ServeHTTP would always
allocate request.Header even if it were not present in the incoming request.
CL 174324 added http.Request.Clone and re-factors ReverseProxy.ServeHTTP
to use the new Clone method. However, the new Clone logic is not equivalent
to the former logic. We preserve former semantics by explicitly allocating
the Header map if nil.
Fixes#33142
Change-Id: I356f94a915dd9779584ce3fe31e56e5474b9ad37
Reviewed-on: https://go-review.googlesource.com/c/go/+/186437
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
There were at least three races in the implementation of the pool of
idle HTTP connections before this CL.
The first race is that HTTP/2 connections can be shared for many
requests, but each requesting goroutine would take the connection out
of the pool and then immediately return it before using it; this
created unnecessary, tiny little race windows during which another
goroutine might dial a second connection instead of reusing the first.
This CL changes the idle pool to just leave the HTTP/2 connection in
the pool permanently (until there is reason to close it), instead of
doing the take-it-out-put-it-back dance race.
The second race is that “is there an idle connection?” and
“register to wait for an idle connection” were implemented as two
separate steps, in different critical sections. So a client could end
up registered to wait for an idle connection and be waiting or perhaps
dialing, not having noticed the idle connection sitting in the pool
that arrived between the two steps.
The third race is that t.getIdleConnCh assumes that the inability to
send on the channel means the client doesn't need the result, when it
could mean that the client has not yet entered the select.
That is, the main dial does:
idleConnCh := t.getIdleConnCh(cm)
select {
case v := <-dialc:
...
case pc := <-idleConnCh
...
...
}
But then tryPutIdleConn does:
waitingDialer := t.idleConnCh[key] // what getIdleConnCh(cm) returned
select {
case waitingDialer <- pconn:
// We're done ...
return nil
default:
if waitingDialer != nil {
// They had populated this, but their dial won
// first, so we can clean up this map entry.
delete(t.idleConnCh, key)
}
}
If the client has returned from getIdleConnCh but not yet reached the
select, tryPutIdleConn will be unable to do the send, incorrectly
conclude that the client does not care anymore, and put the connection
in the idle pool instead, again leaving the client dialing unnecessarily
while a connection sits in the idle pool.
(It's also odd that the success case does not clean up the map entry,
and also that the map has room for only a single waiting goroutine for
a given host.)
None of these races mattered too much before Go 1.11: at most they
meant that connections were not reused quite as promptly as possible,
or a few more than necessary would be created. But Go 1.11 added
Transport.MaxConnsPerHost, which limited the number of connections
created for a given host. The default is 0 (unlimited), but if a user
did explicitly impose a low limit (2 is common), all these misplaced
conns could easily add up to the entire limit, causing a deadlock.
This was causing intermittent timeouts in TestTransportMaxConnsPerHost.
The addition of the MaxConnsPerHost support added its own races.
For example, here t.incHostConnCount could increment the count
and return a channel ready for receiving, and then the client would
not receive from it nor ever issue the decrement, because the select
need not evaluate these two cases in order:
select {
case <-t.incHostConnCount(cmKey):
// count below conn per host limit; proceed
case pc := <-t.getIdleConnCh(cm):
if trace != nil && trace.GotConn != nil {
trace.GotConn(httptrace.GotConnInfo{Conn: pc.conn, Reused: pc.isReused()})
}
return pc, nil
...
}
Obviously, unmatched increments are another way to get to a deadlock.
TestTransportMaxConnsPerHost deadlocked approximately 100% of
the time with a small random sleep added between incHostConnCount
and the select:
ch := t.incHostConnCount(cmKey):
time.Sleep(time.Duration(rand.Intn(10))*time.Millisecond)
select {
case <-ch
// count below conn per host limit; proceed
case pc := <-t.getIdleConnCh(cm):
...
}
The limit also did not properly apply to HTTP/2, because of the
decrement being attached to the underlying net.Conn.Close
and net/http not having access to the underlying HTTP/2 conn.
The alternate decrements for HTTP/2 may also have introduced
spurious decrements (discussion in #29889). Perhaps those
spurious decrements or other races caused the other intermittent
non-deadlock failures in TestTransportMaxConnsPerHost,
in which the HTTP/2 phase created too many connections (#31982).
This CL replaces the buggy, racy code with new code that is hopefully
neither buggy nor racy.
Fixes#29889.
Fixes#31982.
Fixes#32336.
Change-Id: I0dfac3a6fe8a6cdf5f0853722781fe2ec071ac97
Reviewed-on: https://go-review.googlesource.com/c/go/+/184262
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Running the example code when not having permissions
to bind to port 80 will cause the program to hang after
printing the error message.
Change-Id: I2433ba2629b362fc8f1731e40cab5eea72ec354f
GitHub-Last-Rev: 0bb3dc08b6
GitHub-Pull-Request: golang/go#32947
Reviewed-on: https://go-review.googlesource.com/c/go/+/185157
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
CL 46434 changed the doc for Server.IdleTimeout to include
falling back to Server.ReadHeaderTimeout if both
Server.IdleTimeout and Server.ReadTimeout are zero.
However, we explicitly set the ReadDeadlines firstly based
off Server.IdleTimeout or Server.ReadTimeout before attempting
to read the next request, thus the current doc is incorrect.
This CL reverts CL 46434 and also updates the doc for
Server.ReadHeaderTimeout to documenting falling back
to Server.ReadTimeout, if the former is zero, otherwise
there is no timeout.
Fixes#32053
Change-Id: I43dd0252d1bcee6c29a8529abd84c84a49b2fba9
GitHub-Last-Rev: e1cdb59977
GitHub-Pull-Request: golang/go#32164
Reviewed-on: https://go-review.googlesource.com/c/go/+/178337
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
This is the net/http half of #32476. This supplies the method needed
by the other half in x/net/http2 in the already-submitted CL 181259,
which this CL also bundles in h2_bundle.go.
Thanks to Tom Thorogood (@tmthrgd) for the bug report and test.
Fixes#32476
Updates #30694
Change-Id: I79d2a280e486fbf75d116f6695fd3abb61278765
Reviewed-on: https://go-review.googlesource.com/c/go/+/181260
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Roll back CL 161738. That fix changed StripPrefix behavior in the
general case, not just in the situation where where stripping the
prefix from path resulted in the empty string, causing issue #31622.
That kind of change to StripPrefix behavior is not backwards compatible,
and there can be a smaller, more targeted fix for the original issue.
Fixes#31622
Updates #30165
Change-Id: Ie2fcfe6787a32e44f71d564d8f9c9d580fc6f704
Reviewed-on: https://go-review.googlesource.com/c/go/+/180498
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
HTTP 408 responses now exist and are seen in the wild (e.g. from
Google's GFE), so make Go's HTTP client not spam about them when seen.
They're normal (now).
Fixes#32310
Change-Id: I558eb4654960c74cf20db1902ccaae13d03310f6
Reviewed-on: https://go-review.googlesource.com/c/go/+/179457
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
The existing check was introduced to allow tests to pass
on WASM without an environment where the fetch RoundTripper
could run. However, the check now prohibits the use of the
Fetch RoundTripper in all WASM tests, even where the
RoundTripper could run. The new change should only disable
the RoundTripper when used in an environment without fetch.
Fixes#32289
Change-Id: I30d2e0dbcb0e64d4b1a46b583f7e984c2a57d5e5
Reviewed-on: https://go-review.googlesource.com/c/go/+/179118
Run-TryBot: Agniva De Sarker <agniva.quicksilver@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
One of these tests creates a bunch of connections concurrently, then
discovers it doesn't need them all, which then makes the server log
that the client went away midway through the TLS handshake. Perhaps
the server should recognize that as a case not worthy of logging
about, but this is a safer way to eliminate the stderr spam during go
test for now.
The other test's client gives up on its connection and closes it,
similarly confusing the server.
Change-Id: I49ce442c9a63fc437e58ca79f044aa76e8c317b5
Reviewed-on: https://go-review.googlesource.com/c/go/+/179179
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
The typed arrays returned by TypedArrayOf were backed by WebAssembly
memory. They became invalid each time we grow the WebAssembly memory.
This made them very error prone and hard to use correctly.
This change removes TypedArrayOf completely and instead introduces
CopyBytesToGo and CopyBytesToJS for copying bytes between a byte
slice and an Uint8Array. This breaking change is still allowed for
the syscall/js package.
Fixes#31980.
Fixes#31812.
Change-Id: I14c76fdd60b48dd517c1593972a56d04965cb272
Reviewed-on: https://go-review.googlesource.com/c/go/+/177537
Run-TryBot: Richard Musiol <neelance@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Shorten some of the longest tests that run during all.bash.
Removes 7r 50u 21s from all.bash.
After this change, all.bash is under 5 minutes again on my laptop.
For #26473.
Change-Id: Ie0460aa935808d65460408feaed210fbaa1d5d79
Reviewed-on: https://go-review.googlesource.com/c/go/+/177559
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Gerrit is complaining about pushes that affect these files
and forcing people to use -o nokeycheck, which defeats
the point of the check. Hide the keys from this kind of scan
by marking them explicitly as testing keys.
This is a little annoying but better than training everyone
who ever edits one of these test files to reflexively override
the Gerrit check.
The only remaining keys explicitly marked as private instead
of testing are in examples, and there's not much to do
about those. Hopefully they are not edited as much.
Change-Id: I4431592b5266cb39fe6a80b40e742d97da803a0b
Reviewed-on: https://go-review.googlesource.com/c/go/+/178178
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The "After it is called, changing rw.Header will not affect
rw.HeaderMap" claim predates the Result method which changed how the
Recorder should be used.
Fixes#32144Fixes#32136
Change-Id: I95bdfa5ac489ce7b0202824bb5663f4da188e8a7
Reviewed-on: https://go-review.googlesource.com/c/go/+/178058
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
The motivation for doing so is to avoid making inaccurate claims.
Logging may not go to os.Stderr if anyone overrides the log package's
default output via https://godoc.org/log#SetOutput. Saying that
the standard logger is used should be sufficient to explain the
behavior, and users can infer that os.Stderr is used by default,
unless it's changed.
This change is the same as what was applied to http.Server.ErrorLog
documentation in CL 53950.
Change-Id: I32873fc548ceee573f8616b4d49b8a8b98881803
Reviewed-on: https://go-review.googlesource.com/c/go/+/176817
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
For:
http2: track reused connections
https://golang.org/cl/176720 (updates golang/go#31982)
Some x/sys/unix updates come along for the ride too.
I filed #32031 for making the bundling process less difficult and
error-prone in the future.
Change-Id: Ic822080991ffa2d50352c5f613e45648a327cf16
Reviewed-on: https://go-review.googlesource.com/c/go/+/177037
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
In the reverseproxy, replace use (Header).Get, which returns only one value
of a multiple value header, with using the Header map directly. Also fixes
corresponding tests which hid the bug, and adds more tests.
Fixes#30303
Change-Id: Ic9094b5983043460697748759f6dfd95fc111db7
GitHub-Last-Rev: b41038143f
GitHub-Pull-Request: golang/go#30687
Reviewed-on: https://go-review.googlesource.com/c/go/+/166298
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Updates x/net/http2 to git rev 1da14a5a36f220ea3f03470682b737b1dfd5de22 for:
http2: make empty method mean GET
https://golang.org/cl/169557 (Fixesgolang/go#31061)
http2: don't hang a stream if trailers values are not provided
https://golang.org/cl/161958
Change-Id: I628af8c6d07d19e8f19ee37637243f6c242ef3a1
Reviewed-on: https://go-review.googlesource.com/c/go/+/174677
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Add Unwrap methods to types which wrap an underlying error:
"encodinc/csv".ParseError
"encoding/json".MarshalerError
"net/http".transportReadFromServerError
"net".OpError
"net".DNSConfigError
"net/url".Error
"os/exec".Error
"signal/internal/pty".PtyError
"text/template".ExecError
Add os.ErrTemporary. A case could be made for putting this error
value in package net, since no exported error types in package os
include a Temporary method. However, syscall errors returned from
the os package do include this method.
Add Is methods to error types with a Timeout or Temporary method,
making errors.Is(err, os.Err{Timeout,Temporary}) equivalent to
testing the corresponding method:
"context".DeadlineExceeded
"internal/poll".TimeoutError
"net".adrinfoErrno
"net".OpError
"net".DNSError
"net/http".httpError
"net/http".tlsHandshakeTimeoutError
"net/pipe".timeoutError
"net/url".Error
Updates #30322
Updates #29934
Change-Id: I409fb20c072ea39116ebfb8c7534d493483870dc
Reviewed-on: https://go-review.googlesource.com/c/go/+/170037
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
The testcase created a race between the close of the current connection
and the client grabbing a connection for the next request. The client
may receive the current connection which may be closed during its use.
We can have the trasnport close all idle connections thereby forcing the
client to receive a new connection.
Closing idle connections did not handle cleaning up host connection
counts for http/2. We will now decrement the host connection count for
http/2 connections.
Fixes#31784
Change-Id: Iefc0d0d7ed9fa3acd8b4f42004f1579fc1de63fd
Reviewed-on: https://go-review.googlesource.com/c/go/+/174950
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Using password that returns from User.Password() won't work in this case
because password in Userinfo already unescaped. The solution is uses
User.String() to escape password back again and then stringify it to error.
Fixes#31808
Change-Id: I723aafd5a57a5b69f2dd7d3a21b82ebbd4174451
Reviewed-on: https://go-review.googlesource.com/c/go/+/175018
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Treat HTTP/2 connections as an ongoing persistent connection. When we
are told there is no cached connections, cleanup the associated
connection and host connection count.
Fixes#27753
Change-Id: I6b7bd915fc7819617cb5d3b35e46e225c75eda29
Reviewed-on: https://go-review.googlesource.com/c/go/+/140357
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Previously transport was using the hardcoded bufio.defaultBufSize
(4096), limiting throughput and increasing cpu usage when uploading or
downloading large files.
Add options to allow users to configure the buffer sizes as needed.
I tested the maximum benefit of this change by uploading data from
/dev/zero to a server discarding the bytes. Here is an example upload
using the default buffer size:
$ time ./upload 10 https://localhost:8000/
Uploaded 10.00g in 25.13 seconds (407.49m/s)
real 0m25.135s
user 0m5.167s
sys 0m11.643s
With this change, using 128k buffer size:
$ time ./upload 10 https://localhost:8000/
Uploaded 10.00g in 7.93 seconds (1291.51m/s)
real 0m7.935s
user 0m4.517s
sys 0m2.603s
In real world usage the difference will be smaller, depending on the
local and remote storage and the network.
See https://github.com/nirs/http-bench for more info.
Fixes#22618
Change-Id: Iac99ed839c7b95d6dc66602ba8fe1fc5b500c47c
Reviewed-on: https://go-review.googlesource.com/c/go/+/76410
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Ensures that our HTTP/1.X Server properly responds
with a 501 Unimplemented as mandated by the spec at
RFC 7230 Section 3.3.1, which says:
A server that receives a request message with a
transfer coding it does not understand SHOULD
respond with 501 (Unimplemented).
Fixes#30710
Change-Id: I096904e6df053cd1e4b551774cc27523ff3d09f6
Reviewed-on: https://go-review.googlesource.com/c/go/+/167017
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
KeepAlive field used to report the wording "keep-alive period"
which may be misleading. This field does not represent the whole
TCP keepalive time, that is the inactivity period upon which one
endpoint starts probing the other end. But it acctually specifies
the keepalive interval, that is the time between two keepalive
probes.
Fixes#29089
Change-Id: If99b38ba108830d0e5fe527171a2f5c96a3bcde7
Reviewed-on: https://go-review.googlesource.com/c/go/+/155960
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The behavior of Value.String method on non-string JavaScript types has
changed after CL 169757.
Update the implementation of Transport.RoundTrip method to construct the
Response.Status string without relying on result.Get("status").String(),
since that now returns strings like "<number: 200>" instead of "200".
Fixes#31736
Change-Id: I27b3e6cc95aa65fd1918b1400e88478a154aad12
Reviewed-on: https://go-review.googlesource.com/c/go/+/174218
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Richard Musiol <neelance@gmail.com>
Explicitly warn callers that no URL encoding is performed and
that they might need to do it.
Fixes#31577
Change-Id: I52dc3fd2798ba8c3652d4a967b1c5c48eb69f43b
Reviewed-on: https://go-review.googlesource.com/c/go/+/173319
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Though there is variation in the spelling of canceled,
cancellation is always spelled with a double l.
Reference: https://www.grammarly.com/blog/canceled-vs-cancelled/
Change-Id: I240f1a297776c8e27e74f3eca566d2bc4c856f2f
Reviewed-on: https://go-review.googlesource.com/c/go/+/170060
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Transport.DialerAndTLSConfigSupportsHTTP2 was added just earlier
in CL 130256 but we thought of a better name moments after submitting.
ForceAttemptHTTP2 is shorter, more direct, and doesn't constrain what
we can use it with in the future.
Updates #14391
Updates #27011
Change-Id: Ie5fc71bafcbcaa1941b5d49f748b6d710503d477
Reviewed-on: https://go-review.googlesource.com/c/go/+/172299
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Even when a custom TLS config or custom dialer is specified,
enables HTTP/2 if DialerAndTLSConfigSupportsHTTP2 is true.
By this change, avoid automatically enabling HTTP/2 if DialContext is set.
This change also ensures that DefaultTransport still automatically
enable HTTP/2 as discussed in #14391.
Updates #14391Fixes#27011
Change-Id: Icc46416810bee61dbd65ebc96468335030b80573
Reviewed-on: https://go-review.googlesource.com/c/go/+/130256
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Kunpei Sakai <namusyaka@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
A handful of packages were reimplementing IsExported, so use
token.IsExported instead. This caused the deps test to fail for net/rpc.
However, net/rpc deals with Go types, and go/token is light and fairly
low-level in terms of Go tooling packages, so that's okay.
While at it, replace all uses of ast.IsExported with token.IsExported.
This is more consistent, and also means that the import graphs are
leaner. A couple of files no longer need to import go/ast, for example.
We can't get rid of cmd/compile/internal/types.IsExported, as the
compiler can only depend on go/token as of Go 1.4. However, gc used
different implementations in a couple of places, so consolidate the use
of types.IsExported there.
Finally, we can't get rid of the copied IsExported implementation in
encoding/gob, as go/token depends on it as part of a test. That test
can't be an external test either, so there's no easy way to break the
import cycle.
Overall, this removes about forty lines of unnecessary code.
Change-Id: I86a475b7614261e6a7b0b153d5ca02b9f64a7b2d
Reviewed-on: https://go-review.googlesource.com/c/go/+/172037
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This commit adds a KeepAlive field to ListenConfig and uses it
analogously to Dialer.KeepAlive to set TCP KeepAlives per default on
Accept()
Fixes#23378
Change-Id: I57eaf9508c979e7f0e2b8c5dd8e8901f6eb27fd6
GitHub-Last-Rev: e9e035d53e
GitHub-Pull-Request: golang/go#31242
Reviewed-on: https://go-review.googlesource.com/c/go/+/170678
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
A regression was introduced in CL 137335 (5440bfc) that caused FlushInterval
to not be honored until the first Write() call was encountered. This change
starts the flush timer as part of setting up the maxLatencyWriter.
Fixes#31125Fixes#31126
Change-Id: I75325bd926652922219bd1457b2b00ac6d0d41b0
Reviewed-on: https://go-review.googlesource.com/c/go/+/170066
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
MIME sniffing standard defines the RAR signature as 52 61 72 20 1A 07 00.
But this signature is wrong, the RARlab spec defines the 4th byte must
be 0x21 or "!", not 0x20 or " ". Checking a rar file also indicates that:
$ file abc.rar
abc.rar: RAR archive data, v1d, os: Win32
$ head -c 7 abc.rar | od -v -t x1
0000000 52 61 72 21 1a 07 00
0000007
There is also an issue to fix this problem in MIME standard.
See:
- https://www.rarlab.com/technote.htm#rarsign
- https://github.com/whatwg/mimesniff/issues/63Fixes#30926
Change-Id: Id2e2de7ecbf7f44d37ebaf280efd05e4972c5078
Reviewed-on: https://go-review.googlesource.com/c/go/+/167781
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This also updates the vendored-in versions of several packages: 'go
mod vendor' selects a consistent version of each module, but we had
previously vendored an ad-hoc selection of packages.
Notably, x/crypto/hkdf was previously vendored in at a much newer
commit than the rest of x/crypto. Bringing the rest of x/crypto up to
that commit introduced an import of golang.org/x/sys/cpu, which broke
the js/wasm build, requiring an upgrade of x/sys to pick up CL 165749.
Updates #30228
Updates #30241
Updates #25822
Change-Id: I5b3dbc232b7e6a048a158cbd8d36137af1efb711
Reviewed-on: https://go-review.googlesource.com/c/go/+/164623
Reviewed-by: Filippo Valsorda <filippo@golang.org>
net.TCPConn has the ability to send data out using system calls such as
sendfile when the source data comes from an *os.File. However, the way
that I/O has been laid out in the transport means that the File is
actually wrapped behind two outer io.Readers, and as such the TCP stack
cannot properly type-assert the reader, ensuring that it falls back to
genericReadFrom.
This commit does the following:
* Removes transferBodyReader and moves its functionality to a new
doBodyCopy helper. This is not an io.Reader implementation, but no
functionality is lost this way, and it allows us to unwrap one layer
from the body.
* The second layer of the body is unwrapped if the original reader
was wrapped with ioutil.NopCloser, which is what NewRequest wraps the
body in if it's not a ReadCloser on its own. The unwrap operation
passes through the existing body if there's no nopCloser.
Note that this depends on change https://golang.org/cl/163737 to
properly function, as the lack of ReaderFrom implementation otherwise
means that this functionality is essentially walled off.
Benchmarks between this commit and https://golang.org/cl/163862,
incorporating https://golang.org/cl/163737:
linux/amd64:
name old time/op new time/op delta
FileAndServer_1KB/NoTLS-4 53.2µs ± 0% 53.3µs ± 0% ~ (p=0.075 n=10+9)
FileAndServer_1KB/TLS-4 61.2µs ± 0% 60.7µs ± 0% -0.77% (p=0.000 n=10+9)
FileAndServer_16MB/NoTLS-4 25.3ms ± 5% 3.8ms ± 6% -84.95% (p=0.000 n=10+10)
FileAndServer_16MB/TLS-4 33.2ms ± 2% 13.4ms ± 2% -59.57% (p=0.000 n=10+10)
FileAndServer_64MB/NoTLS-4 106ms ± 4% 16ms ± 2% -84.45% (p=0.000 n=10+10)
FileAndServer_64MB/TLS-4 129ms ± 1% 54ms ± 3% -58.32% (p=0.000 n=8+10)
name old speed new speed delta
FileAndServer_1KB/NoTLS-4 19.2MB/s ± 0% 19.2MB/s ± 0% ~ (p=0.095 n=10+9)
FileAndServer_1KB/TLS-4 16.7MB/s ± 0% 16.9MB/s ± 0% +0.78% (p=0.000 n=10+9)
FileAndServer_16MB/NoTLS-4 664MB/s ± 5% 4415MB/s ± 6% +565.27% (p=0.000 n=10+10)
FileAndServer_16MB/TLS-4 505MB/s ± 2% 1250MB/s ± 2% +147.32% (p=0.000 n=10+10)
FileAndServer_64MB/NoTLS-4 636MB/s ± 4% 4090MB/s ± 2% +542.81% (p=0.000 n=10+10)
FileAndServer_64MB/TLS-4 522MB/s ± 1% 1251MB/s ± 3% +139.95% (p=0.000 n=8+10)
darwin/amd64:
name old time/op new time/op delta
FileAndServer_1KB/NoTLS-8 93.0µs ± 5% 96.6µs ±11% ~ (p=0.190 n=10+10)
FileAndServer_1KB/TLS-8 105µs ± 7% 100µs ± 5% -5.14% (p=0.002 n=10+9)
FileAndServer_16MB/NoTLS-8 87.5ms ±19% 10.0ms ± 6% -88.57% (p=0.000 n=10+10)
FileAndServer_16MB/TLS-8 52.7ms ±11% 17.4ms ± 5% -66.92% (p=0.000 n=10+10)
FileAndServer_64MB/NoTLS-8 363ms ±54% 39ms ± 7% -89.24% (p=0.000 n=10+10)
FileAndServer_64MB/TLS-8 209ms ±13% 73ms ± 5% -65.37% (p=0.000 n=9+10)
name old speed new speed delta
FileAndServer_1KB/NoTLS-8 11.0MB/s ± 5% 10.6MB/s ±10% ~ (p=0.184 n=10+10)
FileAndServer_1KB/TLS-8 9.75MB/s ± 7% 10.27MB/s ± 5% +5.26% (p=0.003 n=10+9)
FileAndServer_16MB/NoTLS-8 194MB/s ±16% 1680MB/s ± 6% +767.83% (p=0.000 n=10+10)
FileAndServer_16MB/TLS-8 319MB/s ±10% 963MB/s ± 4% +201.36% (p=0.000 n=10+10)
FileAndServer_64MB/NoTLS-8 180MB/s ±31% 1719MB/s ± 7% +853.61% (p=0.000 n=9+10)
FileAndServer_64MB/TLS-8 321MB/s ±12% 926MB/s ± 5% +188.24% (p=0.000 n=9+10)
Updates #30377.
Change-Id: I631a73cea75371dfbb418c9cd487c4aa35e73fcd
GitHub-Last-Rev: 4a77dd1b80
GitHub-Pull-Request: golang/go#30378
Reviewed-on: https://go-review.googlesource.com/c/go/+/163599
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Make persistConnWriter implement io.ReaderFrom, via an io.Copy on the
underlying net.Conn. This in turn enables it to use OS level
optimizations such as sendfile.
This has been observed giving performance gains even in the absence
of ReaderFrom, more than likely due to the difference in io's default
buffer (32 KB) versus bufio's (4 KB).
Speedups on linux/amd64:
benchmark old MB/s new MB/s speedup
BenchmarkFileAndServer_16MB/NoTLS-4 662.96 2703.74 4.08x
BenchmarkFileAndServer_16MB/TLS-4 552.76 1420.72 2.57x
Speedups on darwin/amd64:
benchmark old MB/s new MB/s speedup
BenchmarkFileAndServer_16MB/NoTLS-8 357.58 1972.86 5.52x
BenchmarkFileAndServer_16MB/TLS-8 346.20 1067.41 3.08x
Updates #30377.
Change-Id: Ic88d4ac254f665223536fcba4d551fc32ae105b6
GitHub-Last-Rev: a6f67cda2e
GitHub-Pull-Request: golang/go#30390
Reviewed-on: https://go-review.googlesource.com/c/go/+/163737
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
The change fixes the following deviations between the existing implementation and the Spec:
1. Using pattern instead of "mask" for assertion and iteration in the Pattern Matching Algorithm.
2. Rename "image/vnd.microsoft.icon" to "image/x-icon" and add another signature for the same.
3. Using named strings instead of hexadecimal representation in "application/zip" and "application/x-rar-compressed".
4. Reordering "sniffSignatures" in accordance with the Spec section "Identifying a resource with an unknown MIME type".
In addition to the above fixes, unit tests for Image MIME type group are added.
Fixes#30570
Change-Id: I97d2ae22b426c3c57bf8efd2ed9396c0be983688
Reviewed-on: https://go-review.googlesource.com/c/go/+/165277
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This adds benchmarks to test file uploads using PUT requests.
It's designed to complement changes https://golang.org/cl/163599 and
https://golang.org/cl/163737, allowing an easy comparison of
performance before and after these changes are applied.
Updates #30377.
Co-authored-by: Emmanuel Odeke <emm.odeke@gmail.com>
Change-Id: Ib8e692c61e1f7957d88c7101669d4f7fb8110c65
GitHub-Last-Rev: 242622b4fc
GitHub-Pull-Request: golang/go#30424
Reviewed-on: https://go-review.googlesource.com/c/go/+/163862
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
The following corner cases for readCookiesTests are tested now:
- An extra cookie delimiter ";" at the end of a Cookie header
- An empty Cookie header
Change-Id: Id8028b448e4182613fb261bf0903efc98cbf4997
Reviewed-on: https://go-review.googlesource.com/c/go/+/164702
Reviewed-by: Volker Dobler <dr.volker.dobler@gmail.com>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
I had been finding these over a year or so, but none were big enough
changes to warrant CLs. They're a handful now, so clean them all up in a
single commit.
The smaller bodies get a bit simpler, but most importantly, the larger
bodies get unindented.
Change-Id: I5707a6fee27d4c9ff9efd3d363af575d7a4bf2aa
Reviewed-on: https://go-review.googlesource.com/c/go/+/165340
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Adds a magic header value that is translated to the
Fetch API redirect option, following existing practices.
Updates #26769
Change-Id: Iaf1c9f710de63ea941a360b73f1b4bb725331a35
Reviewed-on: https://go-review.googlesource.com/c/go/+/164666
Reviewed-by: Richard Musiol <neelance@gmail.com>
Reviewed-by: Agniva De Sarker <agniva.quicksilver@gmail.com>
Run-TryBot: Agniva De Sarker <agniva.quicksilver@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
replaces broken link with a web.archive.org link.
Change-Id: I438536a6ac51d837c30be5df7d3d0caadf65bb95
GitHub-Last-Rev: 0601e4d6b2
GitHub-Pull-Request: golang/go#30523
Reviewed-on: https://go-review.googlesource.com/c/164761
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Comparing errors using DeepEqual breaks if frame information
is added as proposed in Issue #29934.
Updates #29934.
Change-Id: I4ef076e262109a9d6f5b18846129df2535611d71
Reviewed-on: https://go-review.googlesource.com/c/162178
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>
The path of the new stripped URL should also be cleaned. Since an empty path
may cause unexpected errors in some HTTP handlers, e.g. http.ServeFile.
Fixes#30165
Change-Id: Ib44fdce6388b5d62ffbcab5266925ef8f13f26e2
Reviewed-on: https://go-review.googlesource.com/c/161738
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
TestDumpRequest was failing with -count=2 or more
because for test cases that involved mustReadRequest,
the body was created as a *bufio.Reader. DumpRequest
and DumpRequestOut would then read the body until EOF
and would close it after use.
However, on re-runs of the test, the body would
be terminally exhausted and result in an unexpected
error "http: invalid Read on closed Body".
The update to the test cases adds an extra field "GetReq"
which allows us to construct requests per run of the tests
and hence make the test indefinitely re-runnable/idempotent.
"Req" or "GetReq" are mutually exclusive: either one of them
can be set or nil, but not both.
Fixes#26858
Change-Id: Ice3083dac1aa3249da4afc7075cd984eb159530d
Reviewed-on: https://go-review.googlesource.com/c/153377
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
CL 159157 was doing UTF-8 decoding of URLs. URLs aren't really UTF-8,
even if sometimes they are in some contexts.
Instead, only reject ASCII CTLs.
Updates #27302
Updates #22907
Change-Id: Ibd64efa5d3a93263d175aadf1c9f87deb4670c62
Reviewed-on: https://go-review.googlesource.com/c/160178
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This is a more conservative version of the reverted CL 99135 (which
was reverted in CL 137716)
The net/url part rejects URLs with ASCII CTLs from being parsed and
the net/http part rejects writing them if a bogus url.URL is
constructed otherwise.
Updates #27302
Updates #22907
Change-Id: I09a2212eb74c63db575223277aec363c55421ed8
Reviewed-on: https://go-review.googlesource.com/c/159157
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
We shouldn't pollute people's flags with this debugging flag that was
never really meant to be public. It's certainly not documented.
So keep it for now, but don't register it unless it looks like it's in
use (by looking at os.Args). Kinda gross, but less gross than before.
Fixes#28619
Change-Id: I47498948a26a71ff36f9658a6d9dac73fd0a3016
Reviewed-on: https://go-review.googlesource.com/c/154217
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
The name "Callback" does not fit to all use cases of js.Callback.
This commit changes its name to Func. Accordingly NewCallback
gets renamed to FuncOf, which matches ValueOf and TypedArrayOf.
The package syscall/js is currently exempt from Go's compatibility
promise and js.Callback is already affected by a breaking change in
this release cycle. See #28711 for details.
Fixes#28711
Change-Id: I2c380970c3822bed6a3893909672c15d0cbe9da3
Reviewed-on: https://go-review.googlesource.com/c/153559
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Fix unannounced trailers when body is empty and without announced trailers.
Fixes#29031
Change-Id: If49951a42fe56d4be4436a999627db4c2678659d
GitHub-Last-Rev: 3469adc8f5
GitHub-Pull-Request: golang/go#29032
Reviewed-on: https://go-review.googlesource.com/c/151898
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
And remove some unnecessary textproto references. (The net/http
package's CanonicalHeaderKey just calls textproto's
CanonicalMIMEHeaderKey)
Fixes#28894
Change-Id: Ibd277893a168368c593147a2677ad6130870cb88
Reviewed-on: https://go-review.googlesource.com/c/152157
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Packages in vendor/ directories have a "vendor/" path prefix in GOPATH
mode, but intentionally do not in module mode. Since the import path
is embedded in the compiled output, changing that path invalidates
cache entries and causes cmd/go to try to rebuild (and reinstall) the
vendored libraries, which will fail if the directory containing those
libraries is read-only.
If I understood correctly, this is the approach Russ suggested as an
alternative to https://golang.org/cl/136138.
Fixes#27285Fixes#26988
Change-Id: I8a2507fa892b84cde0a803aaa79e460723da572b
Reviewed-on: https://go-review.googlesource.com/c/147443
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
There are three functions that do Connection header write:
1. transport.go/ persistConn.roundTrip
2. transfer.go/ transferWriter.writeHeader
3. request.go/ Request.write
The root cause is roundTrip didn't lookup into request.Close and
transferWriter
didn't take care of extraHeaders.
Fixes#28886
Change-Id: I1d131019c7cd42eb1bcc972c631b7df7511c1f39
Reviewed-on: https://go-review.googlesource.com/c/150722
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
With this change, callbacks returned by syscall/js.NewCallback
get executed synchronously. This is necessary for the APIs of
many JavaScript libraries.
A callback triggered during a call from Go to JavaScript gets executed
on the same goroutine. A callback triggered by JavaScript's event loop
gets executed on an extra goroutine.
Fixes#26045Fixes#27441
Change-Id: I591b9e85ab851cef0c746c18eba95fb02ea9e85b
Reviewed-on: https://go-review.googlesource.com/c/142004
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Scanning through all path patterns is not necessary, since the
paths do not change frequently. Instead, maintain a sorted list
of path prefixes and return the first match.
name old time/op new time/op delta
ServerMatch-12 134ns ± 3% 17ns ± 4% -86.95% (p=0.000 n=19+20)
Change-Id: I15b4483dc30db413321435ee6815fc9bf2bcc546
Reviewed-on: https://go-review.googlesource.com/c/144937
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
There are cases where HTTP message specifies the Trailer header
but not the Transfer-Encoding = chunked. The existing
implementation would return an error in those cases, without
returning also the message itself.
Instead, it would be preferable to let the library user decide when
the message is valid or not.
This change makes the fixTrailer() function not to return an error
and to keep the Trailer value in the Response.Header map but not
populate Response.Trailer.
Fixes#27197
Change-Id: Ic1e96791fde97f31dc5ecb8de05c8e4f49465c2c
Reviewed-on: https://go-review.googlesource.com/c/145398
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Unlike the go resolver, the existing cgo resolver exchanges both DNS A
and AAAA RR queries unconditionally and causes unreasonable connection
setup latencies to applications using the cgo resolver.
This change adds new argument (`network`) in all functions through the
series of calls: from Resolver.internetAddrList to cgoLookupIPCNAME.
Benefit: no redundant DNS calls if certain IP version is used IPv4/IPv6
(no `AAAA` DNS requests if used tcp4, udp4, ip4 network. And vice
versa: no `A` DNS requests if used tcp6, udp6, ip6 network)
Fixes#25947
Change-Id: I39edbd726d82d6133fdada4d06cd90d401e7e669
Reviewed-on: https://go-review.googlesource.com/c/120215
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This adds a crypto/tls.RecordHeaderError.Conn field containing the TLS
underlying net.Conn for non-TLS handshake errors, and then uses it in
the net/http Server to return plaintext HTTP 400 errors when a client
mistakenly sends a plaintext HTTP request to an HTTPS server. This is the
same behavior as Apache.
Also in crypto/tls: swap two error paths to not use a value before
it's valid, and don't send a alert record when a handshake contains a
bogus TLS record (a TLS record in response won't help a non-TLS
client).
Fixes#23689
Change-Id: Ife774b1e3886beb66f25ae4587c62123ccefe847
Reviewed-on: https://go-review.googlesource.com/c/143177
Reviewed-by: Filippo Valsorda <filippo@golang.org>
The previous ReverseProxy change, CL 137335, introduced a bug which could cause
a race and/or a crash.
This reliably crashed before:
$ go test -short -race -v -run=TestReverseProxyFlushInterval -count=20 net/http/httputil
The problem was a goroutine was running http.ResponseWriter.Flush
after the http.Handler's ServeHTTP completed. There was code to
prevent that (a deferred stop call) but the stop call didn't consider
the case where time.AfterFunc had already fired off a new goroutine
but that goroutine hadn't yet scheduled.
Change-Id: I06357908465a3b953efc33e63c70dec19a501adf
Reviewed-on: https://go-review.googlesource.com/c/140977
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
* Rewrite the flushing code to not use a persistent goroutine, which
also simplifies testing.
* Define the meaning of a negative flush interval. Its meaning doesn't
change, but now it's locked in, and then we can use it to optimize
the performance of the non-buffered case to avoid use of an AfterFunc.
* Support (internal-only) special casing of FlushInterval values per
request/response.
* For now, treat Server-Sent Event responses as unbuffered. (or rather,
immediately flushed from the buffer per-write)
Fixes#27816
Change-Id: Ie0f975c997daa3db539504137c741a96d7022665
Reviewed-on: https://go-review.googlesource.com/c/137335
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
And start using it elsewhere in the standard library, removing the
copies in the process.
While at it, rewrite the io.WriteString godoc to be more clear, since it
can now make reference to the defined interface.
Fixes#27946.
Change-Id: Id5ba223c09c19e5fb49815bd3b1bd3254fc786f3
Reviewed-on: https://go-review.googlesource.com/c/139457
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
If an illegal header write is detected, find the first caller outside of
net/http using runtime.CallersFrames and include the call site in the log
message.
Fixes#18761
Change-Id: I92be00ac206c6ebdd60344ad7bf40a7c4c188547
Reviewed-on: https://go-review.googlesource.com/c/130997
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This will clarify that the resources are not completely pushed yet when `Push` returns and that it starts a separate goroutine. This might be implementation dependant but as I believe there is currently only one implementation it should be added to the documentation of the interface which most people will look up first.
Change-Id: Id151c5563fd0c4e611eb1d93b4f64bf747ddf6d4
GitHub-Last-Rev: 1f46eb9a08
GitHub-Pull-Request: golang/go#25025
Reviewed-on: https://go-review.googlesource.com/c/108939
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
WebSockets requires HTTP/1 in practice (no spec or implementations
work over HTTP/2), so if we get an HTTP request that looks like it's
trying to initiate WebSockets, use HTTP/1, like browsers do.
This is part of a series of commits to make WebSockets work over
httputil.ReverseProxy. See #26937.
Updates #26937
Change-Id: I6ad3df9b0a21fddf62fa7d9cacef48e7d5d9585b
Reviewed-on: https://go-review.googlesource.com/c/137437
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
When http2 fails with ErrNoCachedConn the request is retried with body
that has already been read.
Fixes#25009
Change-Id: I51ed5c8cf469dd8b17c73fff6140ab80162bf267
Reviewed-on: https://go-review.googlesource.com/c/131755
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Currently there is no example for http.Handle in the
documentation. This adds an example.
Change-Id: I66ee9983bea1f5237757e1ef4956eae9a056e963
Reviewed-on: https://go-review.googlesource.com/137715
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
A simple grep over the codebase for "the the" which is often
missed by humans.
Change-Id: Ie4b4f07abfc24c73dcd51c8ef1edf4f73514a21c
Reviewed-on: https://go-review.googlesource.com/138335
Reviewed-by: Dave Cheney <dave@cheney.net>
I omitted vendor directories and anything necessary for bootstrapping.
(Tested by bootstrapping with Go 1.4)
Updates #27864
Change-Id: I7d9b68d0372d3a34dee22966cca323513ece7e8a
Reviewed-on: https://go-review.googlesource.com/137856
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
it looks like we should abort trying to configure the http2 transport
again, once it has been configured already.
Otherwise there will be no effect of these checks and changes, as they
will be overridden later again and the disable logic below will have no
effect, too.
So it really looks like we just forgot a return statement here.
Change-Id: Ic99b3bbc662a4e1e1bdbde77681bd1ae597255ad
Reviewed-on: https://go-review.googlesource.com/134795
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The Fetch API returns a null body if there is no response body,
on browsers that support streaming the response body. This
change ensures we check for both undefined and null bodies
before attempting to read the body.
Fixes#27196
Change-Id: I0da86b61284fe394418b4b431495e715a037f335
Reviewed-on: https://go-review.googlesource.com/131236
Reviewed-by: Richard Musiol <neelance@gmail.com>
Run-TryBot: Richard Musiol <neelance@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
cloneHeader duplicates what Header.clone() method is doing. It's
used in a single place, which can be replaced with the use of the
method.
Change-Id: I6e8bbd6c95063f31ca3695f13fa7478873230525
GitHub-Last-Rev: eb08aeae9b
GitHub-Pull-Request: golang/go#27817
Reviewed-on: https://go-review.googlesource.com/136762
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Fixes missing commas where it wasn't immediately apparent whether
"requests" was being used as a verb or a noun.
Change-Id: Ic8c99b4f46475f40a6160d26a3cd11c215940dd5
GitHub-Last-Rev: 1becf6fabe
GitHub-Pull-Request: golang/go#27649
Reviewed-on: https://go-review.googlesource.com/135135
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Change-Id: Id0e2fb2abad5b776ac0ed76e55e36c6b774b5b7a
Reviewed-on: https://go-review.googlesource.com/132278
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The existing documentation of http.Dir is clear in that, if you want to hide
your files and directories that start with a period, you must create
a custom FileSystem. However, there are currently no example on how
to create a custom FileSystem. This commit provides an example.
Fixes#20759
Change-Id: I5a350675536f81412af384d1a316fd6cd6241563
GitHub-Last-Rev: 8b0b644cd0
GitHub-Pull-Request: golang/go#26768
Reviewed-on: https://go-review.googlesource.com/127576
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>